Logstash error - ERROR StatusLogger No log4j2 configuration file found

I always check logstash configuration file using command line before running logstash service.

After up gradation to Logstash 5, I encountered this error.

ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
2017-01-18 14:29:52,358 main ERROR FileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied)
        at java.io.FileOutputStream.open0(Native Method)
        at java.io.FileOutputStream.open(FileOutputStream.java:270)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
        at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:430)
        at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:403)
        at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:73)
        at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:81)
        at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:103)
        at org.apache.logging.log4j.core.appender.RollingFileAppender.createAppender(RollingFileAppender.java:191)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
 
There were couple of mistakes.
1) I was running logstash on command line as a root. It is required to run it as "logstash" user.
2) I was not giving absolute path for --path.settings and -f options. An interesting discussion thread can be followed here - https://discuss.elastic.co/t/logstash-5-not-running/64449/47

So, if you wish to run logstash(ver 5.x) on command line, you should do this:

Test logstash configurations using command line
----------------------------------------------
[psj@desk joshi]# sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/redis-logstash.conf --config.test_and_exit --verbose


Run logstash on command line and not as a service
-----------------------------------------------------------------------------------
[psj@desk joshi]# sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/redis-logstash.conf

Important notes:

1) Make sure that the options that you give for path.settings and configuration file are absolute.
2) Don't add "/" at the end of directory setting for path.settings (i.e. - /etc/logstash)
3) Keep track of the generated logs under /var/log/logstash/logstash-plain.log

You can change the logging level in log4j2 configuration file by setting status to "info" instead of "error" and you will get to know what is the problem!

 [psj@desk joshi]#head /etc/logstash/log4j2.properties
status = info
name = LogstashPropertiesConfig

Now run logstash on command line - e.g.:

[psj@desk joshi]# sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/redis-logstash.conf

Elasticsearch error - Error - Unable to lock JVM Memory: error=12, reason=Cannot allocate memory

I was doing upgrade to latest version of elasticsearch 5.1 and encountered the following errors:

Error - Unsupported major.minor version 52.0

[root@psj Downloads]# java -version
java version "1.6.0_41"
OpenJDK Runtime Environment (IcedTea6 1.13.13) (rhel-1.13.13.1.el6_8-x86_64)
OpenJDK 64-Bit Server VM (build 23.41-b41, mixed mode)

[root@psj Downloads]# service elasticsearch start
Starting elasticsearch: Exception in thread "main" java.lang.UnsupportedClassVersionError: org/elasticsearch/bootstrap/Elasticsearch : Unsupported major.minor version 52.0
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:648)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:272)
        at java.net.URLClassLoader.access$000(URLClassLoader.java:68)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:207)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:201)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:200)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:325)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:296)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:270)
        at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:406)
                                                           [FAILED]


                                                         
After upgrade of JDK to the most recent version of JDK and re-running of elasticsearch rpm, the above error disappeared!

In another case, I once again encountered the following error:

Error - Unable to lock JVM Memory: error=12, reason=Cannot allocate memory

[2017-01-16T12:17:17,524][WARN ][o.e.b.JNANatives         ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2017-01-16T12:17:17,524][WARN ][o.e.b.JNANatives         ] This can result in part of the JVM being swapped out.
[2017-01-16T12:17:17,525][WARN ][o.e.b.JNANatives         ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2017-01-16T12:17:17,526][WARN ][o.e.b.JNANatives         ] These can be adjusted by modifying /etc/security/limits.conf, for example:
        # allow user 'elasticsearch' mlockall
        elasticsearch soft memlock unlimited
        elasticsearch hard memlock unlimited
[2017-01-16T12:17:17,526][WARN ][o.e.b.JNANatives         ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2017-01-16T12:17:17,868][INFO ][o.e.n.Node               ] [elk5-datanode] initializing ...
[2017-01-16T12:17:18,374][INFO ][o.e.e.NodeEnvironment    ] [elk5-datanode] using [1] data paths, mounts [[/ (/dev/mapper/VolGroup-lv_root)]], net usable_space [3gb], net total_space [6.6gb], spins? [possibly], types [ext4]
[2017-01-16T12:17:18,374][INFO ][o.e.e.NodeEnvironment    ] [elk5-datanode] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-01-16T12:17:18,376][INFO ][o.e.n.Node               ] [elk5-datanode] node name [elk5-datanode], node ID [kXxE48JFRxW_AuiHZr1oJQ]
[2017-01-16T12:17:18,379][INFO ][o.e.n.Node               ] [elk5-datanode] version[5.1.1], pid[20670], build[5395e21/2016-12-06T12:36:15.409Z], OS[Linux/2.6.32-431.1.2.el6.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_71/25.71-b15]
[2017-01-16T12:17:49,880][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [aggs-matrix-stats]
[2017-01-16T12:17:49,880][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [ingest-common]
[2017-01-16T12:17:49,880][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [lang-expression]
[2017-01-16T12:17:49,880][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [lang-groovy]
[2017-01-16T12:17:49,882][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [lang-mustache]
[2017-01-16T12:17:49,882][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [lang-painless]
[2017-01-16T12:17:49,882][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [percolator]
[2017-01-16T12:17:49,882][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [reindex]
[2017-01-16T12:17:49,883][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [transport-netty3]
[2017-01-16T12:17:49,883][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [transport-netty4]
[2017-01-16T12:17:49,883][INFO ][o.e.p.PluginsService     ] [elk5-datanode] no plugins loaded
[2017-01-16T12:17:57,751][INFO ][o.e.n.Node               ] [elk5-datanode] initialized
[2017-01-16T12:17:57,752][INFO ][o.e.n.Node               ] [elk5-datanode] starting ...
[2017-01-16T12:17:58,179][INFO ][o.e.t.TransportService   ] [elk5-datanode] publish_address {10.35.2.21:9300}, bound_addresses {10.35.2.21:9300}
[2017-01-16T12:17:58,188][INFO ][o.e.b.BootstrapCheck     ] [elk5-datanode] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-01-16T12:17:58,194][ERROR][o.e.b.Bootstrap          ] [elk5-datanode] node validation exception
bootstrap checks failed
memory locking requested for elasticsearch process but memory is not locked
max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]
[2017-01-16T12:17:58,205][INFO ][o.e.n.Node               ] [elk5-datanode] stopping ...
[2017-01-16T12:17:58,253][INFO ][o.e.n.Node               ] [elk5-datanode] stopped
[2017-01-16T12:17:58,253][INFO ][o.e.n.Node               ] [elk5-datanode] closing ...
[2017-01-16T12:17:58,277][INFO ][o.e.n.Node               ] [elk5-datanode] closed


To get rid of this error, add the following contents to "/etc/security/limits.conf" file as shown below:

[root@psj Downloads]# cat /etc/security/limits.conf |grep -v ^#|grep -v ^$
elasticsearch - memlock unlimited
elasticsearch - nproc 4096
* soft - nofile 65536
* hard - nofile 65536
root soft - nofile 65536
root hard - nofile 65536

Then, create a directory:
# mkdir -p /etc/systemd/system/elasticsearch.service.d

# Create a file elasticsearch.conf in this directory with the following content:
[root@psj Downloads]# cat /etc/systemd/system/elasticsearch.service.d/elasticsearch.conf
[Service]
LimitNPROC=2048

If  you have installed pam, pam provides a file /etc/security/limits.d/90-nproc.conf which explicitly overrides the settings in security.conf for the number of processes.

So, modify the file 90-nproc.conf and specify the number of processes:

[root@psj Downloads]# cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     2048
root       soft    nproc     unlimited

Elasticsearch 5 : max number of threads error

I wanted to test the latest version of Elasticsearch released recently. ELK stack has made the log collection super easy and you can quickly generate a nice working system in a jiffy.

So, I downloaded and installed elasticsearch rpm but it failed to start and gave the following error on my CentOS system:

[2017-01-06T10:41:43,737][INFO ][o.e.b.BootstrapCheck     ] [Jeicebz] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-01-06T10:41:43,745][ERROR][o.e.b.Bootstrap          ] [Jeicebz] node validation exception
bootstrap checks failed
max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]
[2017-01-06T10:41:43,749][INFO ][o.e.n.Node               ] [Jeicebz] stopping ...
[2017-01-06T10:41:43,818][INFO ][o.e.n.Node               ] [Jeicebz] stopped
[2017-01-06T10:41:43,819][INFO ][o.e.n.Node               ] [Jeicebz] closing ...
[2017-01-06T10:41:43,838][INFO ][o.e.n.Node               ] [Jeicebz] closed

I tried to setup the security limits as suggested in elastic portal but it was of no help.

As per my belief, my limits.conf (/etc/security/limits.conf) was OK:

[root@psj joshi]# cat /etc/security/limits.conf |grep -v ^#|grep -v ^$
elasticsearch - memlock unlimited
elasticsearch - nproc 4096
* soft - nofile 32768
* hard - nofile 65536
root soft - nofile 32768
root hard - nofile 65536

Finally, the issue turns out to be systemd related idiocracies :

I have to create a file to override the systemd elasticsearch settings as described below:

[root@psj joshi]# mkdir -p /etc/systemd/system/elasticsearch.service.d

[root@psj joshi]# cat /etc/systemd/system/elasticsearch.service.d/elasticsearch.conf
[Service]
LimitNPROC=2048

If  you have installed pam, pam provides a file /etc/security/limits.d/90-nproc.conf which explicitly overrides the settings in security.conf for the number of processes.

So, modify the file 90-nproc.conf and specify the number of processes:


[root@psj joshi]# cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     2048
root       soft    nproc     unlimited

Now, start elasticsearch once again:
# service elasticsearch restart

and Hurry!! it worked!

[root@psj joshi]# tail -f /var/log/elasticsearch/elasticsearch.log
[2017-01-06T13:06:51,672][INFO ][o.e.b.BootstrapCheck     ] [Jeicebz] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-01-06T13:06:54,794][INFO ][o.e.c.s.ClusterService   ] [Jeicebz] new_master {Jeicebz}{JeicebzWSiCZ3kD88HpXbA}{oBmTUCY5QI-IRHx4q5qSCg}{10.3.2.8}{10.3.2.8:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-01-06T13:06:54,836][INFO ][o.e.h.HttpServer         ] [Jeicebz] publish_address {10.35.21.84:9200}, bound_addresses {10.3.2.8:9200}
[2017-01-06T13:06:54,836][INFO ][o.e.n.Node               ] [Jeicebz] started
[2017-01-06T13:06:55,350][INFO ][o.e.g.GatewayService     ] [Jeicebz] recovered [1] indices into cluster_state

The following links were useful:

1. https://underyx.me/2015/05/18/raising-the-maximum-number-of-file-descriptors
2. https://fredrikaverpil.github.io/2016/04/27/systemd-and-resource-limits/ 
3. https://www.elastic.co/guide/en/elasticsearch/reference/master/setting-system-settings.html
4. http://gerardnico.com/wiki/linux/limits.conf 

Error - can not find "backports.shutil-get-terminal-size" during IPython installation

[root@joshi]# /usr/local/bin/pip2.7 install ipython

During installation, you may encounter this issue:
Collecting backports.shutil-get-terminal-size; python_version == "2.7" (from ipython)
  Could not find a version that satisfies the requirement backports.shutil-get-terminal-size; python_version == "2.7" (from ipython) (from versions: )
No matching distribution found for backports.shutil-get-terminal-size; python_version == "2.7" (from ipython)



 To overcome this issue, download and install "backports.shutil_get_terminal_size" manually and start ipython installation once again:

[root@joshi]# wget https://pypi.python.org/packages/ec/9c/368086faa9c016efce5da3e0e13ba392c9db79e3ab740b763fe28620b18b/backports.shutil_get_terminal_size-1.0.0.tar.gz

 (https://pypi.python.org/pypi/backports.shutil_get_terminal_size/1.0.0)

[root@joshi]# pip2.7 install backports.shutil_get_terminal_size-1.0.0.tar.gz

[root@joshi]# /usr/local/bin/pip2.7 install ipython -i http://pypi.python.org/pypi/simple

Python build error - Python build finished, but the necessary bits to build these modules were not found:

I was compiling python 2.7 from source for Bro and during 'make' step, encountered some issues like:


Python build finished, but the necessary bits to build these modules were not found:
_curses            _curses_panel      _tkinter       
bsddb185           bz2                dl             
imageop            readline           sunaudiodev    
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

Failed to build these modules:
_io                                                  


The 'make' step of compilation was complete but, my attention was towards the above line. It seemed that some important development libraries were missing! After going through setup.py, I found out the reason and installed the development libraries through yum.

Basically do this:,
yum install bzip2-devel
yum install readline-devel
yum install openssl-devel
yum install expat-devel
yum install gdbm-devel
yum install sqlite-devel
A good note on how to compile python is available here - http://anjsimmo.blogspot.in/2014/10/how-to-install-python-34-from-source-on.html

Bro compilation issues: ImportError: No module named _sqlite3 and /usr/local/lib/libpython2.7.a: could not read symbols: Bad value

I was compiling bro-2.4.1 from source and encountered some strange issues during compilation.

The first error I encountered - "No module named sqlite3" even though I have installed python2.7 as requested in bro documentation:

-- Found SubnetTree: build from source aux/pysubnettree 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
    from dbapi2 import *
  File "/usr/local/lib/python2.7/sqlite3/dbapi2.py", line 28, in <module>
    from _sqlite3 import *
ImportError: No module named _sqlite3
CMake Error at aux/broctl/CMakeLists.txt:65 (message):
  The sqlite3 python module is required to use BroControl, but was not found.
  Configuration aborted.

If you have not installed subnettree package, please install it using pip:

#pip install pysubnettree

If you have installed subnettree package and still getting the error then install sqlite3-devel package:

#yum install sqlite-devel

After this, re-compile python2.7 once again:

#./configure --prefix=/usr/local --enable-unicode=ucs4 --enable-shared
# make
# make altinstall
# ln-s /usr/local/bin/python2.7 /usr/bin/python2.7
# ln-s /usr/local/bin/pip2.7 /usr/bin/pip2.7

If you already have python2.6 installed, you can also do:

# cp /usr/lib64/python2.6/lib-dynload/_sqlite3.so /usr/local/lib/python2.7/sqlite3/

Often I miss is setting of environment variables before compilation:
# export LDFLAGS=-L/usr/local/lib
# export CFLAGS=-I/usr/local/include
# export CPPFLAGS=-I/usr/local/include
# export LB_LIBRARY_PATH=-I/usr/local/lib
# export CFLAGS="$CFLAGS -fPIC"

So, don't forget them!!

You may also encounter the following strange error during bro compilation:

/usr/bin/ld: /usr/local/lib/libpython2.7.a(abstract.o): relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a shared object; recompile with -fPIC
/usr/local/lib/libpython2.7.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
make[3]: *** [aux/broctl/aux/pysubnettree/_SubnetTree.so] Error 1
make[3]: Leaving directory `/home/admin/Downloads/src_packages/bro-2.4.1/build'
make[2]: *** [aux/broctl/aux/pysubnettree/CMakeFiles/_SubnetTree.dir/all] Error 2
make[2]: Leaving directory `/home/admin/Downloads/src_packages/bro-2.4.1/build'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/home/admin/Downloads/src_packages/bro-2.4.1/build'
make: *** [all] Error 2

This error occurs when your python version is compiled without using --enable-shared option. So, to avoid this error, use the following:

[psj@ids Python-2.7.10]# ./configure --prefix=/usr/local --enable-unicode=ucs4 --enable-shared

or with LDFLAGS:

[psj@ids Python-2.7.10]# ./configure --prefix=/usr/local --enable-unicode=ucs4 --enable-shared LDFLAGS='-Wl,-rpath /usr/local/lib'

Basically,  re-compile python followed by bro once again -

[psj@ids Python-2.7.10]# ./configure --prefix=/usr/local --enable-unicode=ucs4 --enable-shared LDFLAGS='-Wl,-rpath /usr/local/lib'
[psj@ids Python-2.7.10]# make
[psj@ids Python-2.7.10]# make install

Now, compile bro:
[psj@ids bro-2.4.1]# ./configure

[psj@ids bro-2.4.1]# make
[psj@ids bro-2.4.1]# make install

Some time, you may also have to delete any previous file:
[psj@ids Python-2.7.10]# rm /usr/local/lib/libpython2.7.a
rm: remove regular file `/usr/local/lib/libpython2.7.a'? y


The following links are useful while debugging:
1) https://code.google.com/archive/p/modwsgi/wikis/InstallationIssues.wiki#Mixing_32_Bit_And_64_Bit_Packages
2) https://groups.google.com/forum/#!searchin/modwsgi/$2Fusr$2Flocal$2Flib$2Flibpython2.7.a$3A$20could$20not$20read$20symbols$3A$20Bad$20value/modwsgi/i3-QlhC62WI/dG06OvgkaMQJ

Error - Can't locate ExtUtils/MakeMaker.pm in @INC (@INC

While trying to compile latest version of collectd based metrics monitoring system, I stumbled across the error:

if test ! -d buildperl; then \
      mkdir -p buildperl/Collectd/Plugins; \
      cp ./perl/lib/Collectd.pm buildperl/; \
      cp ./perl/Makefile.PL buildperl/; \
      cp ./perl/lib/Collectd/Unixsock.pm buildperl/Collectd/; \
      cp ./perl/lib/Collectd/Plugins/OpenVZ.pm buildperl/Collectd/Plugins/; \
    fi
touch .perl-directory-stamp
cd buildperl && /usr/bin/perl Makefile.PL PREFIX=NONE
Can't locate ExtUtils/MakeMaker.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at Makefile.PL line 1.
BEGIN failed--compilation aborted at Makefile.PL line 1.

Check whether you have set $PERL_MM_OPT in your environment and try using ./configure --with-perl-bindings=""

The solution was simple! I forgot to install perl.

# yum install perl perl-devel

and vola!!