Redis installation error - /deps/hiredis/libhiredis.a: No such file or directory

During compilation of redis from source, I encountered a strange error:

[root@psj redis-3.2.6]# make
cd src && make all
make[1]: Entering directory `/home/joshi/redis-3.2.6/src'
    LINK redis-server
cc: ../deps/hiredis/libhiredis.a: No such file or directory
cc: ../deps/lua/src/liblua.a: No such file or directory
cc: ../deps/jemalloc/lib/libjemalloc.a: No such file or directory
make[1]: *** [redis-server] Error 1
make[1]: Leaving directory `/home/joshi/redis-3.2.6/src'
make: *** [all] Error 2

To get rid of this error, please install the following packages through yum:

yum install gcc gcc-c++ kernel-devel -y
yum install jemalloc jemalloc-devel -y

Logstash error - ERROR StatusLogger No log4j2 configuration file found

I always check logstash configuration file using command line before running logstash service.

After up gradation to Logstash 5, I encountered this error.

ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
2017-01-18 14:29:52,358 main ERROR FileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied)
        at java.io.FileOutputStream.open0(Native Method)
        at java.io.FileOutputStream.open(FileOutputStream.java:270)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
        at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:430)
        at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:403)
        at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:73)
        at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:81)
        at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:103)
        at org.apache.logging.log4j.core.appender.RollingFileAppender.createAppender(RollingFileAppender.java:191)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
 
There were couple of mistakes.
1) I was running logstash on command line as a root. It is required to run it as "logstash" user.
2) I was not giving absolute path for --path.settings and -f options. An interesting discussion thread can be followed here - https://discuss.elastic.co/t/logstash-5-not-running/64449/47

So, if you wish to run logstash(ver 5.x) on command line, you should do this:

Test logstash configurations using command line
----------------------------------------------
[psj@desk joshi]# sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/redis-logstash.conf --config.test_and_exit --verbose


Run logstash on command line and not as a service
-----------------------------------------------------------------------------------
[psj@desk joshi]# sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/redis-logstash.conf

Important notes:

1) Make sure that the options that you give for path.settings and configuration file are absolute.
2) Don't add "/" at the end of directory setting for path.settings (i.e. - /etc/logstash)
3) Keep track of the generated logs under /var/log/logstash/logstash-plain.log

You can change the logging level in log4j2 configuration file by setting status to "info" instead of "error" and you will get to know what is the problem!

 [psj@desk joshi]#head /etc/logstash/log4j2.properties
status = info
name = LogstashPropertiesConfig

Now run logstash on command line - e.g.:

[psj@desk joshi]# sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/redis-logstash.conf

Elasticsearch error - Error - Unable to lock JVM Memory: error=12, reason=Cannot allocate memory

I was doing upgrade to latest version of elasticsearch 5.1 and encountered the following errors:

Error - Unsupported major.minor version 52.0

[root@psj Downloads]# java -version
java version "1.6.0_41"
OpenJDK Runtime Environment (IcedTea6 1.13.13) (rhel-1.13.13.1.el6_8-x86_64)
OpenJDK 64-Bit Server VM (build 23.41-b41, mixed mode)

[root@psj Downloads]# service elasticsearch start
Starting elasticsearch: Exception in thread "main" java.lang.UnsupportedClassVersionError: org/elasticsearch/bootstrap/Elasticsearch : Unsupported major.minor version 52.0
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:648)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:272)
        at java.net.URLClassLoader.access$000(URLClassLoader.java:68)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:207)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:201)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:200)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:325)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:296)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:270)
        at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:406)
                                                           [FAILED]


                                                         
After upgrade of JDK to the most recent version of JDK and re-running of elasticsearch rpm, the above error disappeared!

In another case, I once again encountered the following error:

Error - Unable to lock JVM Memory: error=12, reason=Cannot allocate memory

[2017-01-16T12:17:17,524][WARN ][o.e.b.JNANatives         ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2017-01-16T12:17:17,524][WARN ][o.e.b.JNANatives         ] This can result in part of the JVM being swapped out.
[2017-01-16T12:17:17,525][WARN ][o.e.b.JNANatives         ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2017-01-16T12:17:17,526][WARN ][o.e.b.JNANatives         ] These can be adjusted by modifying /etc/security/limits.conf, for example:
        # allow user 'elasticsearch' mlockall
        elasticsearch soft memlock unlimited
        elasticsearch hard memlock unlimited
[2017-01-16T12:17:17,526][WARN ][o.e.b.JNANatives         ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2017-01-16T12:17:17,868][INFO ][o.e.n.Node               ] [elk5-datanode] initializing ...
[2017-01-16T12:17:18,374][INFO ][o.e.e.NodeEnvironment    ] [elk5-datanode] using [1] data paths, mounts [[/ (/dev/mapper/VolGroup-lv_root)]], net usable_space [3gb], net total_space [6.6gb], spins? [possibly], types [ext4]
[2017-01-16T12:17:18,374][INFO ][o.e.e.NodeEnvironment    ] [elk5-datanode] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-01-16T12:17:18,376][INFO ][o.e.n.Node               ] [elk5-datanode] node name [elk5-datanode], node ID [kXxE48JFRxW_AuiHZr1oJQ]
[2017-01-16T12:17:18,379][INFO ][o.e.n.Node               ] [elk5-datanode] version[5.1.1], pid[20670], build[5395e21/2016-12-06T12:36:15.409Z], OS[Linux/2.6.32-431.1.2.el6.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_71/25.71-b15]
[2017-01-16T12:17:49,880][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [aggs-matrix-stats]
[2017-01-16T12:17:49,880][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [ingest-common]
[2017-01-16T12:17:49,880][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [lang-expression]
[2017-01-16T12:17:49,880][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [lang-groovy]
[2017-01-16T12:17:49,882][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [lang-mustache]
[2017-01-16T12:17:49,882][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [lang-painless]
[2017-01-16T12:17:49,882][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [percolator]
[2017-01-16T12:17:49,882][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [reindex]
[2017-01-16T12:17:49,883][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [transport-netty3]
[2017-01-16T12:17:49,883][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [transport-netty4]
[2017-01-16T12:17:49,883][INFO ][o.e.p.PluginsService     ] [elk5-datanode] no plugins loaded
[2017-01-16T12:17:57,751][INFO ][o.e.n.Node               ] [elk5-datanode] initialized
[2017-01-16T12:17:57,752][INFO ][o.e.n.Node               ] [elk5-datanode] starting ...
[2017-01-16T12:17:58,179][INFO ][o.e.t.TransportService   ] [elk5-datanode] publish_address {10.35.2.21:9300}, bound_addresses {10.35.2.21:9300}
[2017-01-16T12:17:58,188][INFO ][o.e.b.BootstrapCheck     ] [elk5-datanode] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-01-16T12:17:58,194][ERROR][o.e.b.Bootstrap          ] [elk5-datanode] node validation exception
bootstrap checks failed
memory locking requested for elasticsearch process but memory is not locked
max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]
[2017-01-16T12:17:58,205][INFO ][o.e.n.Node               ] [elk5-datanode] stopping ...
[2017-01-16T12:17:58,253][INFO ][o.e.n.Node               ] [elk5-datanode] stopped
[2017-01-16T12:17:58,253][INFO ][o.e.n.Node               ] [elk5-datanode] closing ...
[2017-01-16T12:17:58,277][INFO ][o.e.n.Node               ] [elk5-datanode] closed


To get rid of this error, add the following contents to "/etc/security/limits.conf" file as shown below:

[root@psj Downloads]# cat /etc/security/limits.conf |grep -v ^#|grep -v ^$
elasticsearch - memlock unlimited
elasticsearch - nproc 4096
* soft - nofile 65536
* hard - nofile 65536
root soft - nofile 65536
root hard - nofile 65536

Then, create a directory:
# mkdir -p /etc/systemd/system/elasticsearch.service.d

# Create a file elasticsearch.conf in this directory with the following content:
[root@psj Downloads]# cat /etc/systemd/system/elasticsearch.service.d/elasticsearch.conf
[Service]
LimitNPROC=2048

If  you have installed pam, pam provides a file /etc/security/limits.d/90-nproc.conf which explicitly overrides the settings in security.conf for the number of processes.

So, modify the file 90-nproc.conf and specify the number of processes:

[root@psj Downloads]# cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     2048
root       soft    nproc     unlimited

Elasticsearch 5 : max number of threads error

I wanted to test the latest version of Elasticsearch released recently. ELK stack has made the log collection super easy and you can quickly generate a nice working system in a jiffy.

So, I downloaded and installed elasticsearch rpm but it failed to start and gave the following error on my CentOS system:

[2017-01-06T10:41:43,737][INFO ][o.e.b.BootstrapCheck     ] [Jeicebz] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-01-06T10:41:43,745][ERROR][o.e.b.Bootstrap          ] [Jeicebz] node validation exception
bootstrap checks failed
max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]
[2017-01-06T10:41:43,749][INFO ][o.e.n.Node               ] [Jeicebz] stopping ...
[2017-01-06T10:41:43,818][INFO ][o.e.n.Node               ] [Jeicebz] stopped
[2017-01-06T10:41:43,819][INFO ][o.e.n.Node               ] [Jeicebz] closing ...
[2017-01-06T10:41:43,838][INFO ][o.e.n.Node               ] [Jeicebz] closed

I tried to setup the security limits as suggested in elastic portal but it was of no help.

As per my belief, my limits.conf (/etc/security/limits.conf) was OK:

[root@psj joshi]# cat /etc/security/limits.conf |grep -v ^#|grep -v ^$
elasticsearch - memlock unlimited
elasticsearch - nproc 4096
* soft - nofile 32768
* hard - nofile 65536
root soft - nofile 32768
root hard - nofile 65536

Finally, the issue turns out to be systemd related idiocracies :

I have to create a file to override the systemd elasticsearch settings as described below:

[root@psj joshi]# mkdir -p /etc/systemd/system/elasticsearch.service.d

[root@psj joshi]# cat /etc/systemd/system/elasticsearch.service.d/elasticsearch.conf
[Service]
LimitNPROC=2048

If  you have installed pam, pam provides a file /etc/security/limits.d/90-nproc.conf which explicitly overrides the settings in security.conf for the number of processes.

So, modify the file 90-nproc.conf and specify the number of processes:


[root@psj joshi]# cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     2048
root       soft    nproc     unlimited

Now, start elasticsearch once again:
# service elasticsearch restart

and Hurry!! it worked!

[root@psj joshi]# tail -f /var/log/elasticsearch/elasticsearch.log
[2017-01-06T13:06:51,672][INFO ][o.e.b.BootstrapCheck     ] [Jeicebz] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-01-06T13:06:54,794][INFO ][o.e.c.s.ClusterService   ] [Jeicebz] new_master {Jeicebz}{JeicebzWSiCZ3kD88HpXbA}{oBmTUCY5QI-IRHx4q5qSCg}{10.3.2.8}{10.3.2.8:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-01-06T13:06:54,836][INFO ][o.e.h.HttpServer         ] [Jeicebz] publish_address {10.35.21.84:9200}, bound_addresses {10.3.2.8:9200}
[2017-01-06T13:06:54,836][INFO ][o.e.n.Node               ] [Jeicebz] started
[2017-01-06T13:06:55,350][INFO ][o.e.g.GatewayService     ] [Jeicebz] recovered [1] indices into cluster_state

The following links were useful:

1. https://underyx.me/2015/05/18/raising-the-maximum-number-of-file-descriptors
2. https://fredrikaverpil.github.io/2016/04/27/systemd-and-resource-limits/ 
3. https://www.elastic.co/guide/en/elasticsearch/reference/master/setting-system-settings.html
4. http://gerardnico.com/wiki/linux/limits.conf