Thursday, 30 April 2015

Logstash - require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar

After installation of logstash 1.4.2 rpm on CentOS, I got stuck across this error:

[root@psj admin]# tail -f /var/log/logstash/logstash.err
  require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
  require at org/jruby/RubyKernel.java:1085
   (root) at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/ffi.rb:1
   (root) at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:1
  require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
  require at org/jruby/RubyKernel.java:1085
     LibC at /opt/logstash/lib/logstash/util/prctl.rb:4
   (root) at /opt/logstash/lib/logstash/util/prctl.rb:3
     main at /opt/logstash/lib/logstash/runner.rb:79
   (root) at /opt/logstash/lib/logstash/runner.rb:215
^C

[root@psj admin]# service logstash status
logstash is not running


After some time, I realized that it may be a path issue  or not getting proper java environment variables issue.
So, to get rid of this, please check your environment for JAVA paths:

[root@psj ELK]# cat /etc/environment
JRE_HOME=/usr/java/jre1.8.0_40/
JAVA_HOME=/usr/java/jdk1.8.0_31
JDK_HOME=/usr/java/jdk1.8.0_31/
[root@psj ELK]# source /etc/environment

Also, add logstash and java to the path:

[root@psj ELK]# export PATH=$PATH:/usr/java/default/bin
[root@psj ELK]# export PATH=/opt/logstash/bin:$PATH

[root@psj ELK]# service logstash status
logstash is running
If this is not your case, perhaps, you can look up bug details reported on github.
https://github.com/elastic/logstash/issues/1289

Logstash - LoadError: Could not load FFI Provider: (NotImplementedError) FFI not available: null

If you get error "LoadError: Could not load FFI Provider" while running logstash(ver 1.4.2) daemon on CentOS like:


[root@psj ELK]# cat /var/log/logstash/logstash.err
LoadError: Could not load FFI Provider: (NotImplementedError) FFI not available: null
 See http://jira.codehaus.org/browse/JRUBY-4583
  require at org/jruby/RubyKernel.java:1085
  require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
   (root) at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/ffi/ffi.rb:69
  require at org/jruby/RubyKernel.java:1085
   (root) at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:1
  require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
  require at org/jruby/RubyKernel.java:1085
   (root) at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/ffi.rb:1
   (root) at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:1
  require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
  require at org/jruby/RubyKernel.java:1085
     LibC at /opt/logstash/lib/logstash/util/prctl.rb:4
   (root) at /opt/logstash/lib/logstash/util/prctl.rb:3
     main at /opt/logstash/lib/logstash/runner.rb:79
   (root) at /opt/logstash/lib/logstash/runner.rb:215


Please do the following:

Modify "LS_JAVA_OPTS" option in /etc/sysconfig/logstash file  as given below:

#vim /etc/sysconfig/logstash
...
#LS_JAVA_OPTS="-Djava.io.tmpdir=$HOME"
#LS_JAVA_OPTS="-Djava.net.preferIPv4Stack=true -Djava.io.tmpdir=$HOME"
LS_JAVA_OPTS="-Djava.net.preferIPv4Stack=true -Djava.io.tmpdir=/var/lib/logstash"
...

"Djava.io.tmpdir" can be set to any directory of your preference instead of "/var/lib/logstash".

[root@psj ELK]# service logstash status
logstash is running

Wednesday, 29 April 2015

DDR Memory timings

I wanted to upgrade system memory in my PC from 4GB to 16GB and got puzzled by the terminologies followed - e.g. DDR3-1333, PC3-12800 by the memory manufacturers.

DDR memory specifications follow DDR-xxx/PC-YYYY classification.

After searching on Google for few seconds, I realized that both the ways refer to memory timings only.

The first number, xxx, indicates the maximum clock speed that the memory chips supports-e.g. DDR-400 memory work at 400 MHz, DDR2-800 can work up to 800 MHz, and DDR3-1333 can work up to 1,333 MHz. It is to be noted that this is not the real clock speed of the memory and the real clock of the DDR, DDR2, and DDR3 memories is usually half of the labeled clock speed.

The second number indicates the maximum theoretical transfer rate that the memory reaches, in MB/s- e.g. DDR-400 memory transfer data at 3,200 MB/s, and are labeled as PC-3200. The DDR2-800 memory transfers data at 6,400 MB/s and are labeled as PC2-6400. Where-as, DDR3-1333 memory can transfer data at 10,664 MB/s and are labeled as PC3-10600 or PC3-10666. The number “2” or “3” after “DDR” or “PC” indicates that we are talking about a DDR2 or DDR3 memory, not DDR.


Maximum memory transfer rate (MB/second) = clock speed (in MHz) * 8.

 i.e.
  • 1066 MHz = PC3-8500 8500 MB/s 
  • 1333 MHz = PC3-10600 10600 MB/s
  • 1600 MHz = PC3-12800 12800 MB/s  
 The following site was useful for finding the memory timings information:
http://www.hardwaresecrets.com/article/understanding-ram-timings/26

Sunday, 19 April 2015

Issue in Elasticsearch Curator cron job

I learned a lesson that cron treats '%' character as  a special character.

I have written "elasticsearch-curator" cronjob for closing the elasticsearch indexes older than 2 days and the elasticsearch index was not getting closed for some reason. However, I was able to execute the same command in bash shell without any issues.

0 8 * * * /usr/bin/curator --host 10.1.0.46 close indices --time-unit days --timestring "%Y.%m.%d" --older-than 2 > /dev/null 2>&1

The google search pointed me to the link:
http://www.ducea.com/2008/11/12/using-the-character-in-crontab-entries/

and then, I realized that I have to escape % character in my cron job!!

0 8 * * * /usr/bin/curator --host 10.1.0.46 close indices --time-unit days --timestring "\%Y.\%m.\%d" --older-than 2 > /dev/null 2>&1

Similarly, if you wish to delete older indices, you can use:

$ /usr/bin/curator --host 10.44.0.46 delete indices --older-than 10 --time-unit days --timestring '%Y.%m.%d' --prefix netflow- 

Friday, 10 April 2015

Installation of OpenAppID pre-processor for Snort IDS

I have heard many good things about OpenAppID pre-processor for snort and wanted to include it in my existing snort installation before up-gradation. This pre-processor allows you to detect application running on your network and can be a great aid in identifying suspicious applications or any application not confirming to your company policy.

There is a good installation note for OpenAppID from snort team but, I felt, there are some missing links. So, here is the sequence that is to be followed for enabling OpenAppID for your snort IDS installation on CentOS or its equivalent linux distribution system:

Make sure that the following rpms are present on the system. If not install them using yum.

# yum install ethtool make zlib zlib-devel gcc gcc-c++ libtool.x86_64 pcre-devel libpcap libpcap-devel flex bison tcpdump autoconf unzip python-setuptools python-devel lua lua-devel


Download snort and its associated libraries from snort site:

All the downloaded packages are saved under /home/admin/install directory.

Now, let us compile and install them one-by-one.

# cd /home/admin/installs
# tar xzvf libdnet-1.12.tar.gz
# cd libdnet-1.12/
# ./configure
# make
# make install
# cd ..
# tar xzvf LuaJIT-2.0.3.tar.gz
# cd LuaJIT-2.0.3/
# make
# make install
# cd ..
# tar -xzvf daq-2.0.4.tar.gz
# cd daq-2.0.4/
# ./configure
# make
# make install
# ldconfig
# cd ..

Now, compile snort with openAppID pre-processor.

# tar -xvf snort-2.9.7.2.tar.gz
# cd snort-2.9.7.2
# ./configure --enable-sourcefire --enable-open-appid
# make
# make install
# which snort
/usr/local/bin/snort
# /usr/local/bin/snort --version

,,_ -*> Snort! <*-
o" )~ Version 2.9.7.2 GRE (Build 177)

Now, configure snort configuration files and create some directories:

# mkdir /etc/snort # For configuration
# mkdir /var/log/snort # For log data
# mkdir /usr/local/lib/snort_dynamicrules # For dynamic rules
# mkdir /etc/snort/rules # For normal text rules
# touch /etc/snort/white_list.rules # For white lists
# touch /etc/snort/black_list.rules # For black lists

A set of configuration files are included in the snort tarball. These files need be copied into /etc/snort/ directory.

# cd /home/admin/installs/snort-2.9.7.2
# cp etc/* /etc/snort/

This process will copy the files - file_magic.conf,snort.conf,unicode.map,classification.config,gen-msg.map,reference.config,threshold.conf to /etc/snort

Now, extract snort registered rules(snort-snapshot-2.9.7.2) and copy them to /etc/snort

# cd /home/admin/installs
# mkdir -p snort_rules
# mv snortrules-snapshot-2.9.7.2.tar.gz snort_rules
# cd snort_rules
# tar -zxvf snortrules-snapshot-2.9.7.2.tar.gz
# cp -r preproc_rules /etc/snort
# cp -r rules /etc/snort
# cp -r so_rules /etc/snort

The next step is configure  snort configuration file - /etc/snort/snort.conf. The following changes are required to be made:

# vim /etc/snort/snort.conf

RULE_PATH /etc/snort/rules
SO_RULE_PATH /etc/snort/so_rules
PREPROC_RULE_PATH /etc/snort/preproc_rules
WHITE_LIST_PATH /etc/snort
BLACK_LIST_PATH /etc/snort




# comment path to dynamic rules libraries
#dynamicdetection directory /usr/local/lib/snort_dynamicrules

The next step is to add the configuration for the OpenAppID preprocessor to the snort.conf file. Find the lines for the reputation preprocessor. Just after the reputation preprocessor and before Step 6 we will add another preprocessor setting.

preprocessor appid: app_stats_filename appstats-u2.log, \
   app_stats_period 60, \
   app_detector_dir /usr/local/snort

This will turn on the OpenAppID preprocessor. The first line names the configuration file to which application statistics will be logged, the second one indicates the time period used to sample this data and the third one specifies the directory which contains the odp directory we extracted from the Open App ID Detector package.

Now, let us configure output section in snort.conf.

Again open snort.conf file and look into Step 6 to find the lines explaining the unified2 output type.

In that section add the following line:
output unified2: filename snort_openappid.log, limit 128, appid_event_types

Now fire up Snort instance:

# /usr/local/snort/bin/snort -c /usr/local/snort/etc/snort.conf -i eth#

Where eth# is which ever interface you will be monitoring with (e.g. eth0).


Possible Errors

1) daq_static library not found:

# ./configure: line 15736: daq-modules-config: command not found
checking for daq_load_modules in -ldaq_static... no

   ERROR!  daq_static library not found, go get it from
   http://www.snort.org/.

This happens if daq_static library is not in the path. So. add "/usr/local/bin" to the path variable.

[root@psj snort-2.9.7.2]# which daq_static
/usr/bin/which: no daq_static in (/sbin:/bin:/usr/sbin:/usr/bin)
[root@psj snort-2.9.7.2]# which daq-modules-config
/usr/bin/which: no daq-modules-config in (/sbin:/bin:/usr/sbin:/usr/bin)
[root@psj snort-2.9.7.2]# export PATH=$PATH:/usr/local/bin


2) LuaJIT library not found:


[root@psj snort-2.9.7.2]# ./configure --enable-sourcefire --enable-open-appid
checking for a BSD-compatible install... /usr/bin/install -c

checking pkg-config is at least version 0.9.0... yes
checking for luajit... no

   ERROR!  LuaJIT library not found. For better performance, go get it from
   http://www.luajit.org/.
configure: error: "Fatal!"

To correct, install  latest version of LuaJIT from http://luajit.org/download.html

3) libluajit-5.1.so.2: cannot open shared object file: No such file or directory

  Loading dynamic preprocessor library /usr/local/lib/snort_dynamicpreprocessor//libsf_reputation_preproc.so... done
  Loading dynamic preprocessor library /usr/local/lib/snort_dynamicpreprocessor//libsf_dce2_preproc.so... done
  Loading dynamic preprocessor library /usr/local/lib/snort_dynamicpreprocessor//libsf_modbus_preproc.so... done
  Loading dynamic preprocessor library /usr/local/lib/snort_dynamicpreprocessor//libsf_appid_preproc.so... ERROR: Failed to load /usr/local/lib/snort_dynamicpreprocessor//libsf_appid_preproc.so: libluajit-5.1.so.2: cannot open shared object file: No such file or directory
Fatal Error, Quitting..

To correct it, do the following:

[root@psj snort-2.9.7.2]# ldd /usr/local/lib/snort_dynamicpreprocessor/libsf_appid_preproc.so
    linux-gate.so.1 =>  (0x007d8000)
    libluajit-5.1.so.2 => not found
    libdnet.1 => /usr/local/lib/libdnet.1 (0x00c79000)
    libpcre.so.0 => /lib/libpcre.so.0 (0x00a7c000)
    libnsl.so.1 => /lib/libnsl.so.1 (0x00d6f000)
    libm.so.6 => /lib/libm.so.6 (0x00667000)
    libcrypto.so.10 => /usr/lib/libcrypto.so.10 (0x007d9000)
    libdl.so.2 => /lib/libdl.so.2 (0x00eef000)
    libsfbpf.so.0 => /usr/local/lib/libsfbpf.so.0 (0x005c5000)
    libpcap.so.1 => /usr/local/lib/libpcap.so.1 (0x00114000)
    libz.so.1 => /lib/libz.so.1 (0x00e07000)
    libpthread.so.0 => /lib/libpthread.so.0 (0x006e5000)
    libc.so.6 => /lib/libc.so.6 (0x00aac000)
    /lib/ld-linux.so.2 (0x002b8000)

# find /usr/ -name libluajit-5.1.so.2 # Check where is the required .so
/usr/local/lib/libluajit-5.1.so.2

[root@psj snort-2.9.7.2]# ls -l /usr/local/lib/libluajit-5.1.so.2
lrwxrwxrwx 1 root root 22 Apr 10 12:31 /usr/local/lib/libluajit-5.1.so.2 -> libluajit-5.1.so.2.0.3

# ldconfig

Again, try to run snort instance:

# snort -c /etc/snort/snort.conf -T


If you wish to write your own OpenAppID plugins or extend/tailor the functionality, technical details are available in this document.

The following article were very useful while installing and configuring OpenAppID:

1) http://blog.snort.org/2014/03/firing-up-openappid.html
2) http://phucnw.blogspot.in/search?q=snort 
3) http://puremonkey2010.blogspot.in/2014/10/snort-customized-appid-lua-script-as.html?m=1
4) https://www.bilgiguvenligi.gov.tr/saldiri-tespit-sistemleri/snort-openappid-ile-uygulama-farkindaligi.html

The following video nicely explains the concepts behind OpenAppID:
1) http://www.irongeek.com/i.php?page=videos/derbycon4/t402-snort-openappid-how-to-build-an-open-source-next-generation-firewall-adam-hogan
2) http://blog.snort.org/2014/06/openappid-training-videos-how-to-create.html

Presentation links:

1) https://www.snort.org/documents/openappid-detection-webinar

2) http://www.centralohioissa.org/wp-content/uploads/2014/07/OpenAppID-ISSA_Rafeeq-Rehman.pdf
3) https://www.snort.org/documents/55