Python build error - Python build finished, but the necessary bits to build these modules were not found:

I was compiling python 2.7 from source for Bro and during 'make' step, encountered some issues like:


Python build finished, but the necessary bits to build these modules were not found:
_curses            _curses_panel      _tkinter       
bsddb185           bz2                dl             
imageop            readline           sunaudiodev    
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

Failed to build these modules:
_io                                                  


The 'make' step of compilation was complete but, my attention was towards the above line. It seemed that some important development libraries were missing! After going through setup.py, I found out the reason and installed the development libraries through yum.

Basically do this:,
yum install bzip2-devel
yum install readline-devel
yum install openssl-devel
yum install expat-devel
yum install gdbm-devel
yum install sqlite-devel
A good note on how to compile python is available here - http://anjsimmo.blogspot.in/2014/10/how-to-install-python-34-from-source-on.html

Bro compilation issues: ImportError: No module named _sqlite3 and /usr/local/lib/libpython2.7.a: could not read symbols: Bad value

I was compiling bro-2.4.1 from source and encountered some strange issues during compilation.

The first error I encountered - "No module named sqlite3" even though I have installed python2.7 as requested in bro documentation:

-- Found SubnetTree: build from source aux/pysubnettree 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
    from dbapi2 import *
  File "/usr/local/lib/python2.7/sqlite3/dbapi2.py", line 28, in <module>
    from _sqlite3 import *
ImportError: No module named _sqlite3
CMake Error at aux/broctl/CMakeLists.txt:65 (message):
  The sqlite3 python module is required to use BroControl, but was not found.
  Configuration aborted.

If you have not installed subnettree package, please install it using pip:

#pip install pysubnettree

If you have installed subnettree package and still getting the error then install sqlite3-devel package:

#yum install sqlite-devel

After this, re-compile python2.7 once again:

#./configure --prefix=/usr/local --enable-unicode=ucs4 --enable-shared
# make
# make altinstall
# ln-s /usr/local/bin/python2.7 /usr/bin/python2.7
# ln-s /usr/local/bin/pip2.7 /usr/bin/pip2.7

If you already have python2.6 installed, you can also do:

# cp /usr/lib64/python2.6/lib-dynload/_sqlite3.so /usr/local/lib/python2.7/sqlite3/

Often I miss is setting of environment variables before compilation:
# export LDFLAGS=-L/usr/local/lib
# export CFLAGS=-I/usr/local/include
# export CPPFLAGS=-I/usr/local/include
# export LB_LIBRARY_PATH=-I/usr/local/lib
# export CFLAGS="$CFLAGS -fPIC"

So, don't forget them!!

You may also encounter the following strange error during bro compilation:

/usr/bin/ld: /usr/local/lib/libpython2.7.a(abstract.o): relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a shared object; recompile with -fPIC
/usr/local/lib/libpython2.7.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
make[3]: *** [aux/broctl/aux/pysubnettree/_SubnetTree.so] Error 1
make[3]: Leaving directory `/home/admin/Downloads/src_packages/bro-2.4.1/build'
make[2]: *** [aux/broctl/aux/pysubnettree/CMakeFiles/_SubnetTree.dir/all] Error 2
make[2]: Leaving directory `/home/admin/Downloads/src_packages/bro-2.4.1/build'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/home/admin/Downloads/src_packages/bro-2.4.1/build'
make: *** [all] Error 2

This error occurs when your python version is compiled without using --enable-shared option. So, to avoid this error, use the following:

[psj@ids Python-2.7.10]# ./configure --prefix=/usr/local --enable-unicode=ucs4 --enable-shared

or with LDFLAGS:

[psj@ids Python-2.7.10]# ./configure --prefix=/usr/local --enable-unicode=ucs4 --enable-shared LDFLAGS='-Wl,-rpath /usr/local/lib'

Basically,  re-compile python followed by bro once again -

[psj@ids Python-2.7.10]# ./configure --prefix=/usr/local --enable-unicode=ucs4 --enable-shared LDFLAGS='-Wl,-rpath /usr/local/lib'
[psj@ids Python-2.7.10]# make
[psj@ids Python-2.7.10]# make install

Now, compile bro:
[psj@ids bro-2.4.1]# ./configure

[psj@ids bro-2.4.1]# make
[psj@ids bro-2.4.1]# make install

Some time, you may also have to delete any previous file:
[psj@ids Python-2.7.10]# rm /usr/local/lib/libpython2.7.a
rm: remove regular file `/usr/local/lib/libpython2.7.a'? y


The following links are useful while debugging:
1) https://code.google.com/archive/p/modwsgi/wikis/InstallationIssues.wiki#Mixing_32_Bit_And_64_Bit_Packages
2) https://groups.google.com/forum/#!searchin/modwsgi/$2Fusr$2Flocal$2Flib$2Flibpython2.7.a$3A$20could$20not$20read$20symbols$3A$20Bad$20value/modwsgi/i3-QlhC62WI/dG06OvgkaMQJ

Error - Can't locate ExtUtils/MakeMaker.pm in @INC (@INC

While trying to compile latest version of collectd based metrics monitoring system, I stumbled across the error:

if test ! -d buildperl; then \
      mkdir -p buildperl/Collectd/Plugins; \
      cp ./perl/lib/Collectd.pm buildperl/; \
      cp ./perl/Makefile.PL buildperl/; \
      cp ./perl/lib/Collectd/Unixsock.pm buildperl/Collectd/; \
      cp ./perl/lib/Collectd/Plugins/OpenVZ.pm buildperl/Collectd/Plugins/; \
    fi
touch .perl-directory-stamp
cd buildperl && /usr/bin/perl Makefile.PL PREFIX=NONE
Can't locate ExtUtils/MakeMaker.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at Makefile.PL line 1.
BEGIN failed--compilation aborted at Makefile.PL line 1.

Check whether you have set $PERL_MM_OPT in your environment and try using ./configure --with-perl-bindings=""

The solution was simple! I forgot to install perl.

# yum install perl perl-devel

and vola!!

Setting up collectd based monitoring system(server with multiple agents) and sending system metrics to graphite

There are many tutorials on setting up collectd as an agent on a machine. However, I have not found  many tutorials that describe how to setup a centralized "collectd" server with multiple collectd agents. This setup will  aggregate system metrics from multiple clients and sends them to Graphite. 
This is my attempt to note down the steps that I used for setting up centralized collectd metric collection system.

Download latest version of collectd.tar.gz (collectd-5.5 or more). Collectd, by default,contains  many plugins e.g. cpu,load,disk,graphite(write_graphite),redis, mysql etc. As a result, it is  possible to capture almost all the system metrics.

Basic installation of collectd involves usual steps:- ./configure and make install for collectd server  as well as collectd agent:

# tar -xvf collectd-version.tar.gz
# cd collectd-version
#./configure --disable-turbostat # if SL <6.6 or CentOS < 6.6
# make
# make install

Copy collectd daemon file to /etc/init.d
# cp ./contrib/redhat/init.d-collectd /etc/init.d/collectd
# chmod +x /etc/init.d/collectd

Make soft links for binaries in /opt/collectd/bin
# ln-s /opt/collectd/sbin/collectdmon /usr/bin/collectdmon
# ln -s /opt/collectd/sbin/collectd /usr/bin/collectd

If you are sending collectd metrics to graphite, please make sure that graphite is installed  and is running prior to compiling collectd. It is presumed that graphite and collectd server  are on the same machine in this use-case.

Setting up collectd server

Once collectd is installed as described above, modify the /opt/collectd/etc/collectd.conf file to contain the following:

Hostname "hostname"
FQDNLookup true
BaseDir "/opt/collectd/var/lib/collectd"
PIDFile "/opt/collectd/var/run/collectd.pid"
PluginDir "/opt/collectd/lib/collectd"
TypesDB "/opt/collectd/share/collectd/types.db"
Interval 10

LoadPlugin logfile
<Plugin logfile>
  LogLevel info
  File "/var/log/collectd.log"
  Timestamp true
  PrintSeverity true
</Plugin>

LoadPlugin network
<Plugin network>
Listen "*" "25826"
</Plugin>

LoadPlugin interface
<Plugin interface>
Interface "eth0"
</Plugin>

LoadPlugin write_graphite
<Plugin write_graphite>
<Node "graphing">
Host "localhost"
Port "2003"
Protocol "tcp"
LogSendErrors true
Prefix "collectd."
StoreRates true
AlwaysAppendDS false
EscapeCharacter "_"
</Node>
</Plugin>

Make adjustments for your network as needed.

Run collectd using:
# service collectd start

Some useful information that is required while setting up:

Default directories for collectd:

Plugins -/opt/collectd/lib/collectd
Binaries - /opt/collectd/sbin/collectd 
Configuration file: /opt/collectd/etc/collectd.conf 

Run collectd like this:
/opt/collectd/sbin/collectd -C /opt/collectd/etc/collectd.conf

Test Collectd configuration:
#collectd -t

Test Collectd plugin configuration:
#collectd -T

Check netstat output:
# nestat -naptul |grep "25826"

Setting up Collectd agent

Now, we are going install the collectd agent on the client machine and then tell it to send the  metrics to the collectd server (not Graphite). The collectd clients do not need "write_graphite"  plugin and can use the older version of Collectd rpms that are available in CentOS/SL  repositories. So, on each client, run:

# yum install collectd collectd-utils

Modify /etc/collectd.conf config file as per your requirement:

Hostname "hostname"
FQDNLookup true
BaseDir "/var/lib/collectd"
PIDFile "/var/run/collectd.pid"
PluginDir "/usr/lib/collectd"
TypesDB "/usr/share/collectd/types.db"
Interval 10
#Timeout 5
ReadThreads 5

LoadPlugin logfile
<Plugin logfile>
  LogLevel info
  File "/var/log/collectd.log"
  Timestamp true
  PrintSeverity true
</Plugin>

LoadPlugin network
<Plugin network>
Server "collectd-server.domain.com" "25826"
</Plugin>

LoadPlugin cpu
LoadPlugin load
LoadPlugin disk
LoadPlugin memory
LoadPlugin processes

Include "/etc/collectd/filters.conf"
Include "/etc/collectd/thresholds.conf"

Be sure to configure the network plugin with your collectd server information.

With this configuration, client metrics statistics are sent to collectd server on port 25826.  These are further sent to Graphite.  If you want to spice up web front-end, you can use grafana and show the trend of system metrics.

Enabling python plugin

If you wish to enable python and iptables plugin support, please do the following:

# yum install python-devel
# yum install iptables-devel

Now, re-compile the collectd source package once again for these modules.
# cd collectd-version
# ./configure --enable-python --enable-iptables
# make
# make install

This process is same for any additional plugins that you may wish to add - e.g. mysql, postgres etc.

Please check the Modules output carefully for the plugin support while configuring source package during compilation.


 Some of the useful links that I encountered while setup:

• https://collectd.org/wiki/index.php
• https://collectd.org/wiki/index.php/Match:Hashed/Config
• http://blog.matthewdfuller.com/2014/06/sending-collectd-metrics-to-graphite.html
• https://keptenkurk.wordpress.com/2015/08/28/using-collectd-with-multiple-networked-instances/
• http://giovannitorres.me/enabling-almost-all-collectd-plugins-on-centos-6.html

Tune your CentOS 6.x system using tuned

tuned, a system performance tuning tool, comes with 9 different system tuning profiles for different scenarios. Each profile implements different tunables for different system resources such as cpu, network, ATA disk.  

tuned, normally runs as daemon and allows dynamic modification of system settings depending on usage.

Basically yo do:
# yum install tuned

#tuned-adm list

Available profiles:
- laptop-ac-powersave
- server-powersave
- laptop-battery-powersave
- desktop-powersave
- virtual-host
- virtual-guest
- enterprise-storage
- throughput-performance
- latency-performance
- spindown-disk
- default

# tuned-adm profile latency-performance

# to turn off:
#tuned-adm off

So, by running tuned, your system will always be optimally tuned.

Installation of flash player on Mozilla firefox in Ubuntu

Before you begin, first check whether Flash is already installed in your system. Visit below official Adobe flash tester page.

Test Your Flash Plugin (https://www.adobe.com/software/flash/about)

In this page, if you see a flash animation and a box mentioning “Version Information” of flash, then it is enabled in Mozilla in your system.

Alternatively, you can also visit about:plugins in Mozilla and check for flash plugin entry.

Now, let's update it as there are many vulnerabilities discovered in Adobe flash in the past year. Also, offline installation is also useful if you are
on internal network. Here  are the steps for manual installation:

1) Download the tar.gz archive from https://get.adobe.com/flashplayer
2) open it with the Archive Manager.
3) Unpack and copy libflashplayer.so to the plugins directory of Firefox -e /home/user/.mozilla/plugins
If it does not exist, create plugins directory and copy file -libflashplayer.so to it.

If "libflashplayer.so" exists in "/usr/lib/adobe-flashplugin/" directory, copy the latest "libflashplayer.so" file to this directory also as it is usually listed first in the path and is picked up by the browser.

And restart the firefox! That's it!

Installing Bro with PF_ring on CentOS 6.x

Bro is an amazing network traffic analysis system. Unfortunately, it is not that popular in information security unlike Snort and frankly, I don't know why!!!

I wanted to install Bro with PF_ring to load balance the traffic on 10G link. Although, the Bro manual details the steps, there are some missing links that took some of my time during installation. So, here are my notes:

Enable/Install EPEL repository
========================
#wget http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6.8.noarch.rpm
#rpm -ivh epel-release-6.8.noarch.rpm

Upgrade cmake
==============
# Cmake rpm available as a part of  CentOS 6/Scientific Linux 6 repository is old. - cmake-2.6.4-5.el6.x86_64

Bro requires that cmake version should be at least 2.8.1 or more - e.g. cmake-2.8.11.2-1.el6.x86_64
This rpm is available as a part of EPEL repository.

Remove existing cmake (ver- 2.6.4)
#yum remove cmake

Install cmake-2.8
#yum install cmake28

Now, make some symbolic links:
#ln -s /usr/bin/cmake28 /usr/bin/cmake
#ln -s /usr/bin/ccmake28 /usr/bin/ccmake
#ln -s /usr/bin/cpack28 /usr/bin/cpack
#ln -s /usr/bin/ctest28 /usr/bin/ctest

Note:

Don't blindly install cmake28 version from EPEL repository like:
#yum install cmake28

This rpm is just a wrapper and has a dependency on cmake26.

So, you should install cmake28-2.8.11.2-1.el6.x86_64 pacakge and not cmake28-2.8.12.2-2.el6.x86_64


ipsumdump installation
======================
#wget http://www.read.seas.harvard.edu/~kohler/ipsumdump
#tar -zxvf ipsumdump-1.85.tar.gz
#cd ipsumdump-1.85
#./configure
#make && make install

#install Bro IDS dependent packages  from linux repository.
===================================
#yum install kernel-devel kernel-headers -y
#yum install make autoconf automake gcc gcc-c++ flex bison libpcap libpcap-devel -y
#yum install openssl openssl-devel python-devel swig zlib zlib-devel -y
#yum install openssl-libs bind-libs -y
#yum install gawk -y
#yum install pcre-devel -y
#yum install libtool -y 
#yum install numactl numactl-devel -y
#yum install gperftools-libs gperftools-devel -y
#yum install GeoIP GeoIP-devel -y
#yum install jemalloc jemalloc-devel -y
#yum install curl -y
#yum install libcurl-devel -y

Set LD flags for python 2.7.10 compilation:

#export LDFLAGS=-L/usr/local/lib
#export CFLAGS=-I/usr/local/include
#export CPPFLAGS=-I/usr/local/include
#export LD_LIBRARY_PATH=/usr/local/lib

Python-2.7.10 installation
==========================
CentOS comes with python2.6 by default. Bro requires python2.7 at least for Broccoli component.
Please do not try to remove existing python version as it will remove many python dependent packages e.g. yum requires python2.6 that comes default with SL/CentOS distribution.

So, install python 2.7.x in addition to existing python 2.6.6

#wget http://www.python.org/ftp/python/2.7.10/Python-2.7.10.tgz
#tar -zxvf Python-2.7.10.tgz
#cd Python-2.7.10
#./configure --prefix=/usr/local --enable-unicode=ucs4 --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"
#make
#make altinstall
#ln -s /usr/local/bin/python2.7 /usr/bin/python2.7

Add python to system path
#export PATH=$PATH:/usr/local/bin/python2.7

If you face any compilation issue,please follow some good blog links that lists python2.7.10 installation instructions:

• http://toomuchdata.com/2014/02/16/how-to-install-python-on-centos/
• https://github.com/h2oai/h2o-2/wiki/Installing-python-2.7-on-centos-6.3.-Follow-this-sequence-exactly-for-centos-machine-only

Now, it is the time to install python package manager - pip so that you can install python packages:
Download file - get-pip.py from https://bootstrap.pypa.io/get-pip.py

# wget get-pip.py from https://bootstrap.pypa.io/get-pip.py 
#python2.7 get-pip.py 

If you are having a local PyPI repository,then

#python2.7 get-pip.py --trusted-host=pypi-local-domain-hostname -i http://local-pypi-repo-url

Now pip will be installed under /usr/local/bin/pip2.7 

Create a symbolic link:
#ln -s /usr/local/bin/pip2.7 /usr/bin/pip2.7

In addition to this, you may be required to install(copy) sqlite3 python bindings on python2.7:

It is presumed that python(python2.6.10) has been installed as a part of default installation.

#cp /usr/lib64/python2.6/lib-dynload/_sqlite3.so /usr/local/lib/python2.7/sqlite3/

Now, install pysubnettree python package:
#pip2.7 install pysubnettree 

On local PyPI:
#pip2.7 install pysubnettree --trusted-host=pypi-local-domain-hostname -i http://local-pypi-repo-url


# Download, install and configure PF_RING
=========================================
Download pf_ring source from http://www.ntop.org/get-started/download/#PF_RING

Now, compile/install various libraries required for PF_RING:

#cd /usr/src
#tar -zxvf PF_RING-6.0.3.tar.gz
#cd PF_RING-6.0.3/userland/lib
#./configure --prefix=/opt/pfring
#make
#make install

#cd ../libpcap
#./configure --prefix=/opt/pfring
#make
#make install

#cd ../tcpdump-4.1.1
#./configure --prefix=/opt/pfring
#make
#make install

#cd ../../kernel

(During kernel 'make' installation step, compile(make) it as normal user rather than as a root.)
#make
#make install

Note - Please make sure that your kernel-devel, kernel-headers and kernel rpms have  same major/minor versions. If not, you will encounter error in make step.
e.g.
# rpm -qa |grep -i kernel
kernel-headers-2.6.32-431.1.2.el6.x86_64
kernel-devel-2.6.32-431.1.2.el6.x86_64
kernel-2.6.32-431.1.2.el6.x86_64

Find out kernel version and try to install corresponding kernel-devel rpm from CentOS/RHEL repository. Do not try to install kernel-devel blindly as there may be a version mismatch between 
kernel-devel and kernel rpms. If not taken care, it will be give you installation headaches!!

Add pf_ring module at start up:

#modprobe pf_ring enable_tx_capture=0 min_num_slots=32768

or

#insmod pf_ring.so enable tx_capture=0 transparent_mode=0 min_num_slots=32768


# Download, install and configure Bro
====================================
Download Bro from bro site - http://www.bro.org/download/index.html
cd bro-2.4.1
./configure --with-pcap=/opt/pfring --enable-debug --enable-perftools --enable-jemalloc
make && make install

To check status of PF_ring
=========================
# modinfo pf_ring
# cat /proc/net/pf_ring/info
# lsmod |grep -i pf_ring

If  you wish to blacklist pf_ring module:
echo "blacklist pf_ring" >> /etc/modprobe.d/blacklist.conf

Once this is done, please follow Bro cluster setup instructions given at :
https://www.bro.org/sphinx/configuration/index.html

Some interesting links for Bro PF_ring installation

•     http://ossectools.blogspot.in/2012/10/multi-node-bro-cluster-setup-howto.html
•     https://thecomputersecurityblog.wordpress.com/2015/03/17/install-bro-on-centos-7-x6-x/
•     http://mailman.icsi.berkeley.edu/pipermail/bro/2013-November/006269.html
•     http://sickbits.net/configuring-a-network-monitoring-system-sensor-w-pf_ring-on-ubuntu-server-1-04-part-1-interface-configuration/
•     https://sathisharthars.wordpress.com/2014/05/07/installing-and-configuring-bro-nids-in-centos-6/
•     https://github.com/h2oai/h2o-2/wiki/Installing-python-2.7-on-centos-6.3.-Follow-this-sequence-exactly-for-centos-machine-only