Monday 16 January 2017

Elasticsearch error - Error - Unable to lock JVM Memory: error=12, reason=Cannot allocate memory

I was doing upgrade to latest version of elasticsearch 5.1 and encountered the following errors:

Error - Unsupported major.minor version 52.0

[root@psj Downloads]# java -version
java version "1.6.0_41"
OpenJDK Runtime Environment (IcedTea6 1.13.13) (rhel-1.13.13.1.el6_8-x86_64)
OpenJDK 64-Bit Server VM (build 23.41-b41, mixed mode)

[root@psj Downloads]# service elasticsearch start
Starting elasticsearch: Exception in thread "main" java.lang.UnsupportedClassVersionError: org/elasticsearch/bootstrap/Elasticsearch : Unsupported major.minor version 52.0
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:648)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:272)
        at java.net.URLClassLoader.access$000(URLClassLoader.java:68)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:207)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:201)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:200)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:325)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:296)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:270)
        at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:406)
                                                           [FAILED]


                                                         
After upgrade of JDK to the most recent version of JDK and re-running of elasticsearch rpm, the above error disappeared!

In another case, I once again encountered the following error:

Error - Unable to lock JVM Memory: error=12, reason=Cannot allocate memory

[2017-01-16T12:17:17,524][WARN ][o.e.b.JNANatives         ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2017-01-16T12:17:17,524][WARN ][o.e.b.JNANatives         ] This can result in part of the JVM being swapped out.
[2017-01-16T12:17:17,525][WARN ][o.e.b.JNANatives         ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2017-01-16T12:17:17,526][WARN ][o.e.b.JNANatives         ] These can be adjusted by modifying /etc/security/limits.conf, for example:
        # allow user 'elasticsearch' mlockall
        elasticsearch soft memlock unlimited
        elasticsearch hard memlock unlimited
[2017-01-16T12:17:17,526][WARN ][o.e.b.JNANatives         ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2017-01-16T12:17:17,868][INFO ][o.e.n.Node               ] [elk5-datanode] initializing ...
[2017-01-16T12:17:18,374][INFO ][o.e.e.NodeEnvironment    ] [elk5-datanode] using [1] data paths, mounts [[/ (/dev/mapper/VolGroup-lv_root)]], net usable_space [3gb], net total_space [6.6gb], spins? [possibly], types [ext4]
[2017-01-16T12:17:18,374][INFO ][o.e.e.NodeEnvironment    ] [elk5-datanode] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-01-16T12:17:18,376][INFO ][o.e.n.Node               ] [elk5-datanode] node name [elk5-datanode], node ID [kXxE48JFRxW_AuiHZr1oJQ]
[2017-01-16T12:17:18,379][INFO ][o.e.n.Node               ] [elk5-datanode] version[5.1.1], pid[20670], build[5395e21/2016-12-06T12:36:15.409Z], OS[Linux/2.6.32-431.1.2.el6.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_71/25.71-b15]
[2017-01-16T12:17:49,880][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [aggs-matrix-stats]
[2017-01-16T12:17:49,880][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [ingest-common]
[2017-01-16T12:17:49,880][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [lang-expression]
[2017-01-16T12:17:49,880][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [lang-groovy]
[2017-01-16T12:17:49,882][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [lang-mustache]
[2017-01-16T12:17:49,882][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [lang-painless]
[2017-01-16T12:17:49,882][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [percolator]
[2017-01-16T12:17:49,882][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [reindex]
[2017-01-16T12:17:49,883][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [transport-netty3]
[2017-01-16T12:17:49,883][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [transport-netty4]
[2017-01-16T12:17:49,883][INFO ][o.e.p.PluginsService     ] [elk5-datanode] no plugins loaded
[2017-01-16T12:17:57,751][INFO ][o.e.n.Node               ] [elk5-datanode] initialized
[2017-01-16T12:17:57,752][INFO ][o.e.n.Node               ] [elk5-datanode] starting ...
[2017-01-16T12:17:58,179][INFO ][o.e.t.TransportService   ] [elk5-datanode] publish_address {10.35.2.21:9300}, bound_addresses {10.35.2.21:9300}
[2017-01-16T12:17:58,188][INFO ][o.e.b.BootstrapCheck     ] [elk5-datanode] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-01-16T12:17:58,194][ERROR][o.e.b.Bootstrap          ] [elk5-datanode] node validation exception
bootstrap checks failed
memory locking requested for elasticsearch process but memory is not locked
max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]
[2017-01-16T12:17:58,205][INFO ][o.e.n.Node               ] [elk5-datanode] stopping ...
[2017-01-16T12:17:58,253][INFO ][o.e.n.Node               ] [elk5-datanode] stopped
[2017-01-16T12:17:58,253][INFO ][o.e.n.Node               ] [elk5-datanode] closing ...
[2017-01-16T12:17:58,277][INFO ][o.e.n.Node               ] [elk5-datanode] closed


To get rid of this error, add the following contents to "/etc/security/limits.conf" file as shown below:

[root@psj Downloads]# cat /etc/security/limits.conf |grep -v ^#|grep -v ^$
elasticsearch - memlock unlimited
elasticsearch - nproc 4096
* soft - nofile 65536
* hard - nofile 65536
root soft - nofile 65536
root hard - nofile 65536

Then, create a directory:
# mkdir -p /etc/systemd/system/elasticsearch.service.d

# Create a file elasticsearch.conf in this directory with the following content:
[root@psj Downloads]# cat /etc/systemd/system/elasticsearch.service.d/elasticsearch.conf
[Service]
LimitNPROC=2048

If  you have installed pam, pam provides a file /etc/security/limits.d/90-nproc.conf which explicitly overrides the settings in security.conf for the number of processes.

So, modify the file 90-nproc.conf and specify the number of processes:

[root@psj Downloads]# cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     2048
root       soft    nproc     unlimited

No comments:

Post a Comment