Elasticsearch 5 : max number of threads error

I wanted to test the latest version of Elasticsearch released recently. ELK stack has made the log collection super easy and you can quickly generate a nice working system in a jiffy.

So, I downloaded and installed elasticsearch rpm but it failed to start and gave the following error on my CentOS system:

[2017-01-06T10:41:43,737][INFO ][o.e.b.BootstrapCheck     ] [Jeicebz] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-01-06T10:41:43,745][ERROR][o.e.b.Bootstrap          ] [Jeicebz] node validation exception
bootstrap checks failed
max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]
[2017-01-06T10:41:43,749][INFO ][o.e.n.Node               ] [Jeicebz] stopping ...
[2017-01-06T10:41:43,818][INFO ][o.e.n.Node               ] [Jeicebz] stopped
[2017-01-06T10:41:43,819][INFO ][o.e.n.Node               ] [Jeicebz] closing ...
[2017-01-06T10:41:43,838][INFO ][o.e.n.Node               ] [Jeicebz] closed

I tried to setup the security limits as suggested in elastic portal but it was of no help.

As per my belief, my limits.conf (/etc/security/limits.conf) was OK:

[root@psj joshi]# cat /etc/security/limits.conf |grep -v ^#|grep -v ^$
elasticsearch - memlock unlimited
elasticsearch - nproc 4096
* soft - nofile 32768
* hard - nofile 65536
root soft - nofile 32768
root hard - nofile 65536

Finally, the issue turns out to be systemd related idiocracies :

I have to create a file to override the systemd elasticsearch settings as described below:

[root@psj joshi]# mkdir -p /etc/systemd/system/elasticsearch.service.d

[root@psj joshi]# cat /etc/systemd/system/elasticsearch.service.d/elasticsearch.conf
[Service]
LimitNPROC=2048

If  you have installed pam, pam provides a file /etc/security/limits.d/90-nproc.conf which explicitly overrides the settings in security.conf for the number of processes.

So, modify the file 90-nproc.conf and specify the number of processes:


[root@psj joshi]# cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     2048
root       soft    nproc     unlimited

Now, start elasticsearch once again:
# service elasticsearch restart

and Hurry!! it worked!

[root@psj joshi]# tail -f /var/log/elasticsearch/elasticsearch.log
[2017-01-06T13:06:51,672][INFO ][o.e.b.BootstrapCheck     ] [Jeicebz] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-01-06T13:06:54,794][INFO ][o.e.c.s.ClusterService   ] [Jeicebz] new_master {Jeicebz}{JeicebzWSiCZ3kD88HpXbA}{oBmTUCY5QI-IRHx4q5qSCg}{10.3.2.8}{10.3.2.8:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-01-06T13:06:54,836][INFO ][o.e.h.HttpServer         ] [Jeicebz] publish_address {10.35.21.84:9200}, bound_addresses {10.3.2.8:9200}
[2017-01-06T13:06:54,836][INFO ][o.e.n.Node               ] [Jeicebz] started
[2017-01-06T13:06:55,350][INFO ][o.e.g.GatewayService     ] [Jeicebz] recovered [1] indices into cluster_state

The following links were useful:

1. https://underyx.me/2015/05/18/raising-the-maximum-number-of-file-descriptors
2. https://fredrikaverpil.github.io/2016/04/27/systemd-and-resource-limits/ 
3. https://www.elastic.co/guide/en/elasticsearch/reference/master/setting-system-settings.html
4. http://gerardnico.com/wiki/linux/limits.conf