Thursday, 30 April 2015

Logstash - require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar

After installation of logstash 1.4.2 rpm on CentOS, I got stuck across this error:

[root@psj admin]# tail -f /var/log/logstash/logstash.err
  require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
  require at org/jruby/RubyKernel.java:1085
   (root) at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/ffi.rb:1
   (root) at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:1
  require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
  require at org/jruby/RubyKernel.java:1085
     LibC at /opt/logstash/lib/logstash/util/prctl.rb:4
   (root) at /opt/logstash/lib/logstash/util/prctl.rb:3
     main at /opt/logstash/lib/logstash/runner.rb:79
   (root) at /opt/logstash/lib/logstash/runner.rb:215
^C

[root@psj admin]# service logstash status
logstash is not running


After some time, I realized that it may be a path issue  or not getting proper java environment variables issue.
So, to get rid of this, please check your environment for JAVA paths:

[root@psj ELK]# cat /etc/environment
JRE_HOME=/usr/java/jre1.8.0_40/
JAVA_HOME=/usr/java/jdk1.8.0_31
JDK_HOME=/usr/java/jdk1.8.0_31/
[root@psj ELK]# source /etc/environment

Also, add logstash and java to the path:

[root@psj ELK]# export PATH=$PATH:/usr/java/default/bin
[root@psj ELK]# export PATH=/opt/logstash/bin:$PATH

[root@psj ELK]# service logstash status
logstash is running
If this is not your case, perhaps, you can look up bug details reported on github.
https://github.com/elastic/logstash/issues/1289

Logstash - LoadError: Could not load FFI Provider: (NotImplementedError) FFI not available: null

If you get error "LoadError: Could not load FFI Provider" while running logstash(ver 1.4.2) daemon on CentOS like:


[root@psj ELK]# cat /var/log/logstash/logstash.err
LoadError: Could not load FFI Provider: (NotImplementedError) FFI not available: null
 See http://jira.codehaus.org/browse/JRUBY-4583
  require at org/jruby/RubyKernel.java:1085
  require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
   (root) at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/ffi/ffi.rb:69
  require at org/jruby/RubyKernel.java:1085
   (root) at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:1
  require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
  require at org/jruby/RubyKernel.java:1085
   (root) at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/ffi.rb:1
   (root) at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:1
  require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
  require at org/jruby/RubyKernel.java:1085
     LibC at /opt/logstash/lib/logstash/util/prctl.rb:4
   (root) at /opt/logstash/lib/logstash/util/prctl.rb:3
     main at /opt/logstash/lib/logstash/runner.rb:79
   (root) at /opt/logstash/lib/logstash/runner.rb:215


Please do the following:

Modify "LS_JAVA_OPTS" option in /etc/sysconfig/logstash file  as given below:

#vim /etc/sysconfig/logstash
...
#LS_JAVA_OPTS="-Djava.io.tmpdir=$HOME"
#LS_JAVA_OPTS="-Djava.net.preferIPv4Stack=true -Djava.io.tmpdir=$HOME"
LS_JAVA_OPTS="-Djava.net.preferIPv4Stack=true -Djava.io.tmpdir=/var/lib/logstash"
...

"Djava.io.tmpdir" can be set to any directory of your preference instead of "/var/lib/logstash".

[root@psj ELK]# service logstash status
logstash is running

Wednesday, 29 April 2015

DDR Memory timings

I wanted to upgrade system memory in my PC from 4GB to 16GB and got puzzled by the terminologies followed - e.g. DDR3-1333, PC3-12800 by the memory manufacturers.

DDR memory specifications follow DDR-xxx/PC-YYYY classification.

After searching on Google for few seconds, I realized that both the ways refer to memory timings only.

The first number, xxx, indicates the maximum clock speed that the memory chips supports-e.g. DDR-400 memory work at 400 MHz, DDR2-800 can work up to 800 MHz, and DDR3-1333 can work up to 1,333 MHz. It is to be noted that this is not the real clock speed of the memory and the real clock of the DDR, DDR2, and DDR3 memories is usually half of the labeled clock speed.

The second number indicates the maximum theoretical transfer rate that the memory reaches, in MB/s- e.g. DDR-400 memory transfer data at 3,200 MB/s, and are labeled as PC-3200. The DDR2-800 memory transfers data at 6,400 MB/s and are labeled as PC2-6400. Where-as, DDR3-1333 memory can transfer data at 10,664 MB/s and are labeled as PC3-10600 or PC3-10666. The number “2” or “3” after “DDR” or “PC” indicates that we are talking about a DDR2 or DDR3 memory, not DDR.


Maximum memory transfer rate (MB/second) = clock speed (in MHz) * 8.

 i.e.
  • 1066 MHz = PC3-8500 8500 MB/s 
  • 1333 MHz = PC3-10600 10600 MB/s
  • 1600 MHz = PC3-12800 12800 MB/s  
 The following site was useful for finding the memory timings information:
http://www.hardwaresecrets.com/article/understanding-ram-timings/26

Sunday, 19 April 2015

Issue in Elasticsearch Curator cron job

I learned a lesson that cron treats '%' character as  a special character.

I have written "elasticsearch-curator" cronjob for closing the elasticsearch indexes older than 2 days and the elasticsearch index was not getting closed for some reason. However, I was able to execute the same command in bash shell without any issues.

0 8 * * * /usr/bin/curator --host 10.1.0.46 close indices --time-unit days --timestring "%Y.%m.%d" --older-than 2 > /dev/null 2>&1

The google search pointed me to the link:
http://www.ducea.com/2008/11/12/using-the-character-in-crontab-entries/

and then, I realized that I have to escape % character in my cron job!!

0 8 * * * /usr/bin/curator --host 10.1.0.46 close indices --time-unit days --timestring "\%Y.\%m.\%d" --older-than 2 > /dev/null 2>&1

Similarly, if you wish to delete older indices, you can use:

$ /usr/bin/curator --host 10.44.0.46 delete indices --older-than 10 --time-unit days --timestring '%Y.%m.%d' --prefix netflow- 

Friday, 10 April 2015

Installation of OpenAppID pre-processor for Snort IDS

I have heard many good things about OpenAppID pre-processor for snort and wanted to include it in my existing snort installation before up-gradation. This pre-processor allows you to detect application running on your network and can be a great aid in identifying suspicious applications or any application not confirming to your company policy.

There is a good installation note for OpenAppID from snort team but, I felt, there are some missing links. So, here is the sequence that is to be followed for enabling OpenAppID for your snort IDS installation on CentOS or its equivalent linux distribution system:

Make sure that the following rpms are present on the system. If not install them using yum.

# yum install ethtool make zlib zlib-devel gcc gcc-c++ libtool.x86_64 pcre-devel libpcap libpcap-devel flex bison tcpdump autoconf unzip python-setuptools python-devel lua lua-devel


Download snort and its associated libraries from snort site:

All the downloaded packages are saved under /home/admin/install directory.

Now, let us compile and install them one-by-one.

# cd /home/admin/installs
# tar xzvf libdnet-1.12.tar.gz
# cd libdnet-1.12/
# ./configure
# make
# make install
# cd ..
# tar xzvf LuaJIT-2.0.3.tar.gz
# cd LuaJIT-2.0.3/
# make
# make install
# cd ..
# tar -xzvf daq-2.0.4.tar.gz
# cd daq-2.0.4/
# ./configure
# make
# make install
# ldconfig
# cd ..

Now, compile snort with openAppID pre-processor.

# tar -xvf snort-2.9.7.2.tar.gz
# cd snort-2.9.7.2
# ./configure --enable-sourcefire --enable-open-appid
# make
# make install
# which snort
/usr/local/bin/snort
# /usr/local/bin/snort --version

,,_ -*> Snort! <*-
o" )~ Version 2.9.7.2 GRE (Build 177)

Now, configure snort configuration files and create some directories:

# mkdir /etc/snort # For configuration
# mkdir /var/log/snort # For log data
# mkdir /usr/local/lib/snort_dynamicrules # For dynamic rules
# mkdir /etc/snort/rules # For normal text rules
# touch /etc/snort/white_list.rules # For white lists
# touch /etc/snort/black_list.rules # For black lists

A set of configuration files are included in the snort tarball. These files need be copied into /etc/snort/ directory.

# cd /home/admin/installs/snort-2.9.7.2
# cp etc/* /etc/snort/

This process will copy the files - file_magic.conf,snort.conf,unicode.map,classification.config,gen-msg.map,reference.config,threshold.conf to /etc/snort

Now, extract snort registered rules(snort-snapshot-2.9.7.2) and copy them to /etc/snort

# cd /home/admin/installs
# mkdir -p snort_rules
# mv snortrules-snapshot-2.9.7.2.tar.gz snort_rules
# cd snort_rules
# tar -zxvf snortrules-snapshot-2.9.7.2.tar.gz
# cp -r preproc_rules /etc/snort
# cp -r rules /etc/snort
# cp -r so_rules /etc/snort

The next step is configure  snort configuration file - /etc/snort/snort.conf. The following changes are required to be made:

# vim /etc/snort/snort.conf

RULE_PATH /etc/snort/rules
SO_RULE_PATH /etc/snort/so_rules
PREPROC_RULE_PATH /etc/snort/preproc_rules
WHITE_LIST_PATH /etc/snort
BLACK_LIST_PATH /etc/snort




# comment path to dynamic rules libraries
#dynamicdetection directory /usr/local/lib/snort_dynamicrules

The next step is to add the configuration for the OpenAppID preprocessor to the snort.conf file. Find the lines for the reputation preprocessor. Just after the reputation preprocessor and before Step 6 we will add another preprocessor setting.

preprocessor appid: app_stats_filename appstats-u2.log, \
   app_stats_period 60, \
   app_detector_dir /usr/local/snort

This will turn on the OpenAppID preprocessor. The first line names the configuration file to which application statistics will be logged, the second one indicates the time period used to sample this data and the third one specifies the directory which contains the odp directory we extracted from the Open App ID Detector package.

Now, let us configure output section in snort.conf.

Again open snort.conf file and look into Step 6 to find the lines explaining the unified2 output type.

In that section add the following line:
output unified2: filename snort_openappid.log, limit 128, appid_event_types

Now fire up Snort instance:

# /usr/local/snort/bin/snort -c /usr/local/snort/etc/snort.conf -i eth#

Where eth# is which ever interface you will be monitoring with (e.g. eth0).


Possible Errors

1) daq_static library not found:

# ./configure: line 15736: daq-modules-config: command not found
checking for daq_load_modules in -ldaq_static... no

   ERROR!  daq_static library not found, go get it from
   http://www.snort.org/.

This happens if daq_static library is not in the path. So. add "/usr/local/bin" to the path variable.

[root@psj snort-2.9.7.2]# which daq_static
/usr/bin/which: no daq_static in (/sbin:/bin:/usr/sbin:/usr/bin)
[root@psj snort-2.9.7.2]# which daq-modules-config
/usr/bin/which: no daq-modules-config in (/sbin:/bin:/usr/sbin:/usr/bin)
[root@psj snort-2.9.7.2]# export PATH=$PATH:/usr/local/bin


2) LuaJIT library not found:


[root@psj snort-2.9.7.2]# ./configure --enable-sourcefire --enable-open-appid
checking for a BSD-compatible install... /usr/bin/install -c

checking pkg-config is at least version 0.9.0... yes
checking for luajit... no

   ERROR!  LuaJIT library not found. For better performance, go get it from
   http://www.luajit.org/.
configure: error: "Fatal!"

To correct, install  latest version of LuaJIT from http://luajit.org/download.html

3) libluajit-5.1.so.2: cannot open shared object file: No such file or directory

  Loading dynamic preprocessor library /usr/local/lib/snort_dynamicpreprocessor//libsf_reputation_preproc.so... done
  Loading dynamic preprocessor library /usr/local/lib/snort_dynamicpreprocessor//libsf_dce2_preproc.so... done
  Loading dynamic preprocessor library /usr/local/lib/snort_dynamicpreprocessor//libsf_modbus_preproc.so... done
  Loading dynamic preprocessor library /usr/local/lib/snort_dynamicpreprocessor//libsf_appid_preproc.so... ERROR: Failed to load /usr/local/lib/snort_dynamicpreprocessor//libsf_appid_preproc.so: libluajit-5.1.so.2: cannot open shared object file: No such file or directory
Fatal Error, Quitting..

To correct it, do the following:

[root@psj snort-2.9.7.2]# ldd /usr/local/lib/snort_dynamicpreprocessor/libsf_appid_preproc.so
    linux-gate.so.1 =>  (0x007d8000)
    libluajit-5.1.so.2 => not found
    libdnet.1 => /usr/local/lib/libdnet.1 (0x00c79000)
    libpcre.so.0 => /lib/libpcre.so.0 (0x00a7c000)
    libnsl.so.1 => /lib/libnsl.so.1 (0x00d6f000)
    libm.so.6 => /lib/libm.so.6 (0x00667000)
    libcrypto.so.10 => /usr/lib/libcrypto.so.10 (0x007d9000)
    libdl.so.2 => /lib/libdl.so.2 (0x00eef000)
    libsfbpf.so.0 => /usr/local/lib/libsfbpf.so.0 (0x005c5000)
    libpcap.so.1 => /usr/local/lib/libpcap.so.1 (0x00114000)
    libz.so.1 => /lib/libz.so.1 (0x00e07000)
    libpthread.so.0 => /lib/libpthread.so.0 (0x006e5000)
    libc.so.6 => /lib/libc.so.6 (0x00aac000)
    /lib/ld-linux.so.2 (0x002b8000)

# find /usr/ -name libluajit-5.1.so.2 # Check where is the required .so
/usr/local/lib/libluajit-5.1.so.2

[root@psj snort-2.9.7.2]# ls -l /usr/local/lib/libluajit-5.1.so.2
lrwxrwxrwx 1 root root 22 Apr 10 12:31 /usr/local/lib/libluajit-5.1.so.2 -> libluajit-5.1.so.2.0.3

# ldconfig

Again, try to run snort instance:

# snort -c /etc/snort/snort.conf -T


If you wish to write your own OpenAppID plugins or extend/tailor the functionality, technical details are available in this document.

The following article were very useful while installing and configuring OpenAppID:

1) http://blog.snort.org/2014/03/firing-up-openappid.html
2) http://phucnw.blogspot.in/search?q=snort 
3) http://puremonkey2010.blogspot.in/2014/10/snort-customized-appid-lua-script-as.html?m=1
4) https://www.bilgiguvenligi.gov.tr/saldiri-tespit-sistemleri/snort-openappid-ile-uygulama-farkindaligi.html

The following video nicely explains the concepts behind OpenAppID:
1) http://www.irongeek.com/i.php?page=videos/derbycon4/t402-snort-openappid-how-to-build-an-open-source-next-generation-firewall-adam-hogan
2) http://blog.snort.org/2014/06/openappid-training-videos-how-to-create.html

Presentation links:

1) https://www.snort.org/documents/openappid-detection-webinar

2) http://www.centralohioissa.org/wp-content/uploads/2014/07/OpenAppID-ISSA_Rafeeq-Rehman.pdf
3) https://www.snort.org/documents/55

Tuesday, 24 March 2015

Comparing RPM versions

I wanted to compare rpms installed on my system with the Scientific Linux/CentOS repository rpms and update any existing rpm if there is large version difference. This is not usually needed as yum will automatically update the packages to its latest version! But, I was interested in finding out how a 'newer version' is determined when 'rpm -U' or 'yum localupdate' is executed.

After some google searches, I came across rpmdev-vercmp tool. This python based tool is a part of rpmdevtools rpm package. This tool requires that you should know epoch, version and release information for each rpm you need to compare.

You can find out this information for any package using:

#rpm -qa --queryformat "'%{NAME}' '%{EPOCH}:%{VERSION}' '%{RELEASE}' '%{ARCH}'\n" |grep package_name

If you wish to know all the query tags, use the following command:
#rpm --querytags

This information extracted needs to be compared with latest rpms from linux repositories and rpmdev-vercmp tool comes in handy. Otherwise, you have to make string comparisons and there is a chance that you might miss a use-case in your code!! So, I decided to use rpmdev-vercmp utility without hesitation.

$ rpmdev-vercmp --help

rpmdev-vercmp <epoch1> <ver1> <release1> <epoch2> <ver2> <release2>
rpmdev-vercmp <EVR1> <EVR2>
rpmdev-vercmp # with no arguments, prompt

Exit status is 0 if the EVR's are equal, 11 if EVR1 is newer, and 12 if EVR2
is newer.  Other exit statuses indicate problems.

$ rpmdev-vercmp audit-2.2-2.el6.i686 audit-2.2-2.el6.i686
These are equal



Useful links:
  1. http://www.faqssys.info/bash-script-to-verify-that-an-rpm-is-at-least-at-a-given-version/
  2. http://utcc.utoronto.ca/~cks/space/blog/linux/RPMShellVersionComparison

Tuesday, 3 March 2015

Installation of BRO IDS on CentOS

I am using snort IDS for a long time and it generates a lot of useful alerts for malicious activities on my PC. Further, I have heard good things about BRO IDS and wanted to give a try. Bro offers a network analysis framework that is different from the typical IDS like snort.

Here are the steps for installation on CentOS 6.5 or higher linux machines:

1) # Install runtime dependencies.
# yum -y install libpcap openssl-libs bind-libs zlib bash python libcurl gawk GeoIP gperftools-libs

2) # Install the build dependencies.
# yum -y install libpcap-devel openssl-devel bind-devel zlib-devel cmake git perl libcurl-devel GeoIP-devel python-devel gperftools-devel swig

You may also require these libraries and so, install them in advance especially if you are compiling Bro from source tar.gz:

#yum -y install cmake make gcc gcc-c++ flex bison libpcap-devel openssl-devel #yum -y python-devel swig zlib-devel
# yum install jemalloc
# yum install jemalloc-devel
# yum install curl
# yum install libcurl-devel
# yum install GeoIP
# yum install GeoIP-devel
# yum install gperftools
# yum install ruby


3) Install  EPEL repository on the machine.
4) Download and install rpm from Bro site- https://www.bro.org/download/index.html

Of course, if you wish, you can compile the Bro IDS from the source!!

By default, all Bro IDS related files are installed in /opt/bro.

5) Modify default path:
# export path /opt/bro/bin:$PATH

You can also add PATH=/opt/bro2/bin:$PATH to your ~/.profile file in your home directory to make the change permanent.

6) For basic configuration steps, please follow the documentation on the project page:

Using your favorite editor, please modify the following 3 files:
$ PREFIX  refers to the base of bro installation directory
  •     $PREFIX/etc/node.cfg -> Configure the network interface to monitor (i.e. interface=eth0)

[admin@ids]$  cd /opt/bro
[admin@ids bro]$ cat etc/node.cfg
# Example BroControl node configuration.
#
# This example has a standalone node ready to go except for possibly changing
# the sniffing interface.

# This is a complete standalone configuration.  Most likely you will
# only need to change the interface.
[bro]
type=standalone
host=localhost
interface=eth0

  •     $PREFIX/etc/networks.cfg -> Configure the local networks (i.e. 10.0.0.0/8 Private IP space )
[admin@ids bro]$ cat etc/networks.cfg
# List of local networks in CIDR notation, optionally followed by a
# descriptive tag.
# For example, "10.0.0.0/8" or "fe80::/64" are valid prefixes.

10.0.0.0/8          Private IP space
192.168.0.0/16      Private IP space
  •     $PREFIX/etc/broctl.cfg -> Change the MailTo address and the log rotation
 The broctl.cfg file is where in recipient address for all emails send out by Bro and BroControl, and log rotation intervals among other features can be configured.

When you run bro for the first time, a warning may be reported. Please ignore.
# broctl
warning: cannot read '/var/bro/spool/broctl.dat' (this is ok on first run)

Welcome to BroControl 1.2

Type "help" for help.

[BroControl] >


# broctl
warning: cannot read '/var/bro/spool/broctl.dat' (this is ok on first run)

Welcome to BroControl 1.2

Type "help" for help.

[BroControl] >

 [root@ids bro]# broctl

Welcome to BroControl 1.3

Type "help" for help.

[BroControl] > help

BroControl Version 1.3

  capstats [<nodes>] [<secs>]      - Report interface statistics with capstats
  check [<nodes>]                  - Check configuration before installing it
  cleanup [--all] [<nodes>]        - Delete working dirs (flush state) on nodes
  config                           - Print broctl configuration
  cron [--no-watch]                - Perform jobs intended to run from cron
  cron enable|disable|?            - Enable/disable "cron" jobs
  df [<nodes>]                     - Print nodes' current disk usage
  diag [<nodes>]                   - Output diagnostics for nodes
  exec <shell cmd>                 - Execute shell command on all hosts
  exit                             - Exit shell
  install                          - Update broctl installation/configuration
  netstats [<nodes>]               - Print nodes' current packet counters
  nodes                            - Print node configuration
  peerstatus [<nodes>]             - Print status of nodes' remote connections
  print <id> [<nodes>]             - Print values of script variable at nodes
  process <trace> [<op>] [-- <sc>] - Run Bro (with options and scripts) on trace
  quit                             - Exit shell
  restart [--clean] [<nodes>]      - Stop and then restart processing
  scripts [-c] [<nodes>]           - List the Bro scripts the nodes will load
  start [<nodes>]                  - Start processing
  status [<nodes>]                 - Summarize node status
  stop [<nodes>]                   - Stop processing
  top [<nodes>]                    - Show Bro processes ala top
  update [<nodes>]                 - Update configuration of nodes on the fly
 
Commands provided by plugins:

  ps.bro [<nodes>]                 - Show Bro processes on nodes' systems


[BroControl] > cron enable
cron enabled
[BroControl] > install
creating policy directories ... done.
installing site policies ... done.
generating standalone-layout.bro ... done.
generating local-networks.bro ... done.
generating broctl-config.bro ... done.
updating nodes ... done.
[BroControl] >

[BroControl] > status
Name         Type       Host          Status    Pid    Peers  Started
bro          standalone localhost     stopped  
[BroControl] > start
starting bro ...
[BroControl] > status
Name         Type       Host          Status    Pid    Peers  Started
bro          standalone localhost     running   32206  0      04 Mar 12:21:47
[BroControl] >

 That's all!! Check the docs for more information.