Cryptic python subprocess error - OSError: [Errno 2] No such file or directory

Tshark has become swiss army knief for me and I was experimenting with tshark to extract DNS traffic. A portion of the code is reproduced here:

command = /usr/bin/tshark -i ens33 -nn -T fields -e frame.time -e ip.src -e ip.dst -e dns.count.queries -e dns.count.answers -e dns.qry.name -e dns.qry.type -e dns.resp.name -e dns.resp.type -e dns.resp.ttl -e dns.a -e dns.ns -e dns.mx.mail_exchange -e dns.cname -e dns.txt -e dns.flags.rcode -Y 'dns.count.answers gt 0' -E separator='|'

Traceback (most recent call last):
  File "collect.py", line 59, in <module>
    main()
  File "collect.py", line 45, in main
    tshark_response = subprocess.Popen(command, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory


After reading the documentation (http://docs.python.org/2/library/subprocess.html#frequently-used-arguments), I realized that "shell=True" should be added as argument and seems one of the easiest option and Vola! It worked.

Ubuntu Xenial - apt update - E: Some index files failed to download. They have been ignored, or old ones used instead.

Recently, I installed Ubuntu 16.04 on a new PC and updated my repository URLs in "/etc/apt/sources.list" file to point to internal Ubuntu repository servers.


While I was updating my package indexes, I encountered the following errors:

E: Failed to fetch http://repo.xxx.xxx.in/ubuntu/dists/xenial/main/i18n/Translation-en  404  Not Found
E: Failed to fetch http://repo.xxx.xxx.in/ubuntu/dists/xenial-updates/main/i18n/Translation-en  404  Not Found
E: Failed to fetch http://repo.xxx.xxx.in/ubuntu/dists/xenial-backports/main/i18n/Translation-en  404  Not Found
E: Failed to fetch http://repo.xxx.xxx.in/ubuntu/dists/xenial-security/main/i18n/Translation-en  404  Not Found
E: Some index files failed to download. They have been ignored, or old ones used instead.

After searching on google and stackoverflow, it seems a language translation issue. To get rid of this issue, please follow the following steps:

1) Create a file named /etc/apt/apt.conf.d/99translations & add the following content to it:

Acquire::Languages "none";

2) You may also need to remove existing translation files in /var/lib/apt/lists/
$ sudo rm -vf /var/lib/apt/lists/*

3) Now, do cleanup and update of package indexes.

$ sudo apt-get clean
$ sudo apt-get update

Some useful references on stackoverflow:
1) https://askubuntu.com/questions/74653/how-can-i-remove-the-translation-entries-in-apt
2) https://askubuntu.com/questions/762273/16-04-upgrade-failed-to-fetch-empty-files-cant-be-valid-archives/764463

pip based python packages installation through proxy on Windows

For installing python packages through pip via proxy on Windows platform:

1) Create a directory: c:\Users\<User name>\pip
2) Then, create a file "pip.conf" in this directory with the following contents:

[global]
trusted-host = pypi.python.org
proxy = http://user:xxx@192.168.1.4:3128

Now, you are ready to install any python packages through pip:
c:\> pip install requests


Useful references:
1) https://stackoverflow.com/questions/28278207/python-cant-find-pip-ini-or-pip-conf-in-windows7
2) https://stackoverflow.com/questions/14149422/using-pip-behind-a-proxy

Netflow records - TCP flags explained

After setting up nfdump for netflow analysis, I was going through the flow records and got puzzled by flags column. It took some time to understand the concept.


Netflow records contain a field reporting cumulative OR-ed TCP flags seen in the flow. e.g. In a regular TCP connection, a client would sent a SYN, then an ACK, then optional flags like PSH and finally FIN. The table below reports a summary of TCP flags with their binary and decimal values.

Consider two netflow records - a normal TCP connection and closed port

    1) Normal TCP connection - If we sum (using OR operator) all the flags used in a TCP connection (SYN(2) + ACK(16) + PSH(8) + FIN(1)), we have 27.
    Client-to-server: TCP flags = 2 SYN(2)
    Server-to-client: TCP flags = 16 ACK(16)
    Client side Optional flags: TCP flags = 8 PSH(8)

    Client-to-server: TCP flags = 1 FIN(1)

     Cumulative flag value - 2 [SYN(2)] + 16 [ACK(16] + 8 [PSH(8)]+ 1 [FIN(1)]

    2) Closed port - Our client would send a SYN packet while server will reply with RST/ACK packet.So, we have two flows:

    Client-to-server: TCP flags = 2 (SYN(2))
    Server-to-client: TCP flags = 20 (RST(4) + ACK(16))


Good explanation of TCP flags in netflow based on flow-viewer is available here : - https://blog.pierky.com/netflow-weird-tcp-flags-in-flowviewer-and-flow-print/

Elasticsearch - node validation exception bootstrap checks failed on CentOS 6.x

During setting up of elasticsearch 5.x cluster, I encountered "bootstrap checks failed" error as given below:

[2017-02-09T09:47:32,004][ERROR][o.e.b.Bootstrap] [elk5-master-node] node validation exception
bootstrap checks failed
system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2017-02-09T09:47:32,022][INFO ][o.e.n.Node               ] [elk5-master-node] stopping ...
[2017-02-09T09:47:32,163][INFO ][o.e.n.Node               ] [elk5-master-node] stopped
[2017-02-09T09:47:32,163][INFO ][o.e.n.Node               ] [elk5-master-node] closing ...
[2017-02-09T09:47:32,181][INFO ][o.e.n.Node               ] [elk5-master-node] closed


After going through github issue(s), I realized that "secomp" on CentOS 6 is not available and you have to disable it.

There is a discussion on it here -  https://github.com/elastic/elasticsearch/issues/22899

So, add the following settings in elasticsearch.yml:
bootstrap.system_call_filter: false

and start elasticsearch again.

Redis installation error - /deps/hiredis/libhiredis.a: No such file or directory

During compilation of redis from source, I encountered a strange error:

[root@psj redis-3.2.6]# make
cd src && make all
make[1]: Entering directory `/home/joshi/redis-3.2.6/src'
    LINK redis-server
cc: ../deps/hiredis/libhiredis.a: No such file or directory
cc: ../deps/lua/src/liblua.a: No such file or directory
cc: ../deps/jemalloc/lib/libjemalloc.a: No such file or directory
make[1]: *** [redis-server] Error 1
make[1]: Leaving directory `/home/joshi/redis-3.2.6/src'
make: *** [all] Error 2

To get rid of this error, please install the following packages through yum:

yum install gcc gcc-c++ kernel-devel -y
yum install jemalloc jemalloc-devel -y

Logstash error - ERROR StatusLogger No log4j2 configuration file found

I always check logstash configuration file using command line before running logstash service.

After up gradation to Logstash 5, I encountered this error.

ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
2017-01-18 14:29:52,358 main ERROR FileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied)
        at java.io.FileOutputStream.open0(Native Method)
        at java.io.FileOutputStream.open(FileOutputStream.java:270)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
        at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:430)
        at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:403)
        at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:73)
        at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:81)
        at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:103)
        at org.apache.logging.log4j.core.appender.RollingFileAppender.createAppender(RollingFileAppender.java:191)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
 
There were couple of mistakes.
1) I was running logstash on command line as a root. It is required to run it as "logstash" user.
2) I was not giving absolute path for --path.settings and -f options. An interesting discussion thread can be followed here - https://discuss.elastic.co/t/logstash-5-not-running/64449/47

So, if you wish to run logstash(ver 5.x) on command line, you should do this:

Test logstash configurations using command line
----------------------------------------------
[psj@desk joshi]# sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/redis-logstash.conf --config.test_and_exit --verbose


Run logstash on command line and not as a service
-----------------------------------------------------------------------------------
[psj@desk joshi]# sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/redis-logstash.conf

Important notes:

1) Make sure that the options that you give for path.settings and configuration file are absolute.
2) Don't add "/" at the end of directory setting for path.settings (i.e. - /etc/logstash)
3) Keep track of the generated logs under /var/log/logstash/logstash-plain.log

You can change the logging level in log4j2 configuration file by setting status to "info" instead of "error" and you will get to know what is the problem!

 [psj@desk joshi]#head /etc/logstash/log4j2.properties
status = info
name = LogstashPropertiesConfig

Now run logstash on command line - e.g.:

[psj@desk joshi]# sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/redis-logstash.conf