Process dump the attacker using logstash

Use linux "auditd" to monitor /etc/passwd file and generate a key "password_access".

$ sudo apt install auditd
$ auditctl -w /etc/passwd -p wa -k passwd_access

Force logstash core dump any process that causes auditd to write the "password_access" key.

Install gdb (gcore):
sudo apt install gdb

Modify the output section of your /etc/logstash/conf.d/00-output.conf:

output {
  if [key] == "password_access" {
      stdout {codec => json}
      exec { command => "gcore -o /tmp/dump-%{@timestamp} %{pid}"}
  }
}

Why you should not run multiple anti-virus products

I liked the advice from Emsisoft Anti-virus about why you should not run multiple anti-virus products.

* Potential conflicts/incompatibility issues

Modern anti-virus/anti-malware software acts like an extra protection layer that sits between the base of the operating system and the apps/programs that run on it. Developing this type of software is always challenging and requires many years of experience.Protection programs are created in different ways and it may cause unexpected crashes or freezes that are very difficult to resolve.

* Who will quarantine first

Since anti-virus products have real-time scanning enabled, it's a race between multiple anti-virus programs about quarantining potential malicious download/file scan and this may give unexpected results/errors.

* High resource usage

Since number of viruses/malwares are growing exponentially, size and complexity of anti-virus programs is also correspondingly growing and these now consume lot of CPU/storage resources. The situation will be further compounded if you use multiple anti-virus products.

So, in nutshell, avoid installing multiple anti-virus/anti-malware products as it’s not worth it. If you are happy with existing anti-virus software, stick with it.If you are unhappy with it, un-install it and then install a new one.

Ref -
* Don't run multiple anti-virus products - https://blog.emsisoft.com/2017/12/18/do-not-run-multiple-antivirus
* Latest independent tests of anti-virus products - https://www.av-comparatives.org/

Fix "this webpage has a redirect loop" error in browser

On a daily basis, Internet as a whole suffers from many problems and one such problem that is sometimes encountered is

"The webpage has a redirect loop"

The other times, the redirect error manifests itself with the following details:

Error 310 (net::ERR_TOO_MANY_REDIRECTS)

As a result, every time you visit Gmail page in Firefox/Chrome, the redirection loop will prevent access to it.

When Chrome or Firefox starts complaining about redirect loops just do three things:

1) Check the clock and set appropriate Date/Time as per your timezone
2) Clear the Browser cache
3) Kill the browser settings and reset it to default state

By the way, the no of redirect vary depending on browser and are listed below:

• Chrome 64bit version: 49, ↷ Version 62.0.3202.52 (Official Build) beta, 21 redirects
• Chrome Canary 64bit, version: 49 ↷ 63.0.3239.6 (Official Build) canary, 21 redirects
• Firefox 32-bit version: 43 ↷ 56.0, 20 redirects
• Firefox 64-bit version: 43 ↷ 56.0, 20 redirects
• IE version: 8 11 redirects via webpagetest.org
• IE version: 9 121 redirects via webpagetest.org
• IE version: 10 121 redirects via webpagetest.org
• IE version: 11.0.9600.18792 110 redirects
• Opera version: 28, ↷ 48.0.2685.35 (PGO) 21 redirects
• Safari version: 5.1.7, 16 redirects
• Google Nexus 5, Samsung Galaxy S4..S8, Galaxy Tab 4, 21 redirects

The latest firefox version - Quantum supports up to 40 redirects.

Ref -
https://stackoverflow.com/questions/9384474/in-chrome-how-many-redirects-are-too-many

Save bash command history to syslog

# Increase history size
export HISTSIZE=5000

# In the commands given below - every time a new prompt is issued , bash history is appended to the file, then it is cleared from the current shell's memory,  and current shell reloads the history from the file.

$ export PROMPT_COMMAND="history -a; history -c; history -r; ${PROMPT_COMMAND}"

Another option is to export bash commands to syslog where the bash logs can be centralized and analyzed on demand.

Add the following snipplet to bashrc.

[root@psj]# vim /etc/bashrc

PROMPT_COMMAND=$(history -a)
typeset -r PROMPT_COMMAND

function log2syslog
{
   declare command
   command=$BASH_COMMAND
   logger -p local1.notice -t bash -i -- "$USER : $PWD : $command"

}
trap log2syslog DEBUG

Download gz file using python request module

Here is the quick script I wrote to download a gz file using python requests module:

#!/usr/bin/env python
import requests
import gzip
import logging
import sys
import StringIO
import zlib

# setup logging
logging.basicConfig(stream = sys.stdout, level = logging.ERROR)
log = logging.getLogger('threat-feeds-logger')

#proxy configuration
proxy_host='10.1.1.11'
proxy_port=3128
proxy_user = 'xxxx'
proxy_password = 'xxxx'

feed_url = 'https://foo.org/foo.gz'
proxy_dict = {
                'http':'http://%s:%s@%s:%s' % (proxy_user, proxy_password, proxy_host, proxy_port),
                'https':'http://%s:%s@%s:%s' % (proxy_user, proxy_password, proxy_host, proxy_port)
            }
try:
    response = requests.get(feed_url,proxies = proxy_dict)
except Exception as e:
    log.error("Error while getting data from url - %s" %feed_url)

if response.status_code == 200:
    buf_data = StringIO.StringIO(response.content)
    f = gzip.GzipFile(fileobj=buf_data)
    for row in f.readlines():
       print row
 

How to find google IP address range

I wanted to track google IP range in one of the investigation. So, I did this:

[admin@psj ~]$ nslookup -type=txt _spf.google.com 8.8.8.8
Server:        8.8.8.8
Address:    8.8.8.8#53

Non-authoritative answer:
_spf.google.com    text = "v=spf1 include:_netblocks.google.com include:_netblocks2.google.com include:_netblocks3.google.com ~all"

Authoritative answers can be found from:
google.com    nameserver = ns2.google.com.
google.com    nameserver = ns3.google.com.
google.com    nameserver = ns1.google.com.
google.com    nameserver = ns4.google.com.
ns2.google.com    internet address = 216.239.34.10
ns3.google.com    internet address = 216.239.36.10
ns1.google.com    internet address = 216.239.32.10
ns4.google.com    internet address = 216.239.38.10

Run a nslookup for each one:
$ nslookup -q=TXT _netblocks.google.com 8.8.8.8
$ nslookup -q=TXT _netblocks2.google.com 8.8.8.8


[admin@psj ~]$ nslookup -type=txt _netblocks.google.com
Server:        8.8.8.8
Address:    8.8.8.8#53

Non-authoritative answer:
_netblocks.google.com    text = "v=spf1 ip4:64.18.0.0/20 ip4:64.233.160.0/19 ip4:66.102.0.0/20 ip4:66.249.80.0/20 ip4:72.14.192.0/18 ip4:74.125.0.0/16 ip4:108.177.8.0/21 ip4:173.194.0.0/16 ip4:207.126.144.0/20 ip4:209.85.128.0/17 ip4:216.58.192.0/19 ip4:216.239.32.0/19 ~all"

Authoritative answers can be found from:
google.com    nameserver = ns2.google.com.
google.com    nameserver = ns3.google.com.
google.com    nameserver = ns4.google.com.
google.com    nameserver = ns1.google.com.
ns2.google.com    internet address = 216.239.34.10
ns3.google.com    internet address = 216.239.36.10
ns1.google.com    internet address = 216.239.32.10
ns4.google.com    internet address = 216.239.38.10

You can also use whois query to find out network blocks assigned to google.
$ whois 74.125.127.108

Using Bro Threat intelligence framework

** Download a malicious pcap from internet, say, sample1.pcap

** Ensure that you have installed bro correctly and bro intelligence frameworks is present with required modules(Minimum version of bro - 2.4 or greater)

[root@ joshi]# ls -l /usr/local/bro/share/bro/policy/frameworks/intel/
do_notice.bro  seen/
If you want to see if intelligence framework scripts are loaded or not at runtime, kindly look in loaded_scripts.log

** Create a intelligence feed file manually, say, intel1.dat:
(Please note that the separator is TAB and not spaces for columns in the file.)

[root@ joshi]# cat intel1.txt
#fields indicator       indicator_type  meta.source     meta.url        meta.do_notice
172.16.88.10    Intel::ADDR     ciarmy  http://www.ciarmy.com/list/ci-badguys.txt       T


** Create a file: intel.bro in say - /home/joshi with the following contents:

# intelligence framework
@load frameworks/intel/seen
@load frameworks/intel/do_notice

redef Intel::read_files += {
    "/home/joshi/intel1.txt"
};

Another variation:
[root@ joshi]# cat intel1.bro
# intelligence framework
@load frameworks/intel/seen
@load frameworks/intel/do_notice

redef Intel::read_files += {
#    "/home/joshi/intel1.txt"
 @DIR + "/intel1.txt",
};

Now, run bro to make use of "intel1.bro" for finding any malicious connections in the pcap.

[root@ joshi]# bro -C -r /home/joshi/Downloads/sample1.pcap intel1.bro


If you wish, you can also add the above lines to site policy script(local.bro) under /usr/local/bro/share/bro/site/local.bro instead of intel1.bro

By default, the logs will be created in the current directory.

To run local script based on site policy(local.bro):
[root@ joshi]# bro -C -r /home/joshi/Downloads/sample1.pcap local

[root@ joshi]# ls -l *.log
-rw-r--r-- 1 root root 60758 Oct  5 17:38 conn.log
-rw-r--r-- 1 root root     0 Oct  5 17:38 debug.log
-rw-r--r-- 1 root root 25454 Oct  5 17:38 dns.log
-rw-r--r-- 1 root root  3736 Oct  5 17:38 http.log
-rw-r--r-- 1 root root  1547 Oct  5 17:38 intel.log
-rw-r--r-- 1 root root 23896 Oct  5 17:36 loaded_scripts.log
-rw-r--r-- 1 root root  2690 Oct  5 17:38 notice.log
-rw-r--r-- 1 root root   253 Oct  5 17:38 packet_filter.log
-rw-r--r-- 1 root root   384 Oct  5 17:38 weird.log

[root@ joshi]# head intel.log
#separator \x09
#set_separator  ,
#empty_field    (empty)
#unset_field    -
#path   intel
#open   2017-10-05-17-38-26
#fields ts      uid     id.orig_h       id.orig_p       id.resp_h       id.resp_p       fuid file_mime_type   file_desc       seen.indicator  seen.indicator_type     seen.where      seen.node     sources
#types  time    string  addr    port    addr    port    string  string  string  string  enum enum     string  set[string]
1394196008.595180       CQRmeTHpuDAbRlDrf       172.16.88.10    49493   172.16.88.135   80   --       -       172.16.88.135   Intel::ADDR     Conn::IN_RESP   bro     ciarmy
1394196043.661031       Ch4sqx4DcNd8kxx5hj      172.16.88.10    49495   172.16.88.135   80   --       -       172.16.88.135   Intel::ADDR     Conn::IN_RESP   bro     ciarmy


Local networks on command line:
[root@ joshi]# bro -C -r /home/joshi/Downloads/sample1.pcap local "Site::local_nets += {10.0.0.0/8,192.168.0.0/16}"

Note: "-C" command line flag for bro is a MUST. I have to spend few hours to understand the behaviour and finally, it was discovered in bro faq(http://www.bro.org/documentation/faq.html)

It allows bro's event engine to process the packets event if packets don't have valid checksums.

The following links are useful if you want additional information:
*  Bro Intel framework - https://www.bro.org/sphinx-git/frameworks/intel.html
*  PCAP files - https://github.com/aboutsecurity/Bro-samples/
*  http://blog.bro.org/2014/01/intelligence-data-and-bro_4980.html

Installing yara

Installation of yara
-----------------------
$ wget https://github.com/VirusTotal/yara/archive/v3.5.0.tar.gz
$ cd yara-3.5.0
$ ./bootstrap.sh
$ ./configure
$ make
# to spot any errors
$ make check
# make install
yara documentation is very good and you can find installation instructions on variety of
platforms. Please refer - http://yara.readthedocs.io/en/latest/gettingstarted.html#

Test yara with your own rule!
$ echo "rule dummy { condition: true }" > my_rule
$ yara -r my_rule my_rule
dummy my_rule

Yara links
----------
Yara repository - https://github.com/VirusTotal/yara/archive/v3.5.0.tar.gz
yara rules - https://github.com/Yara-Rules/rules
https://bruteforce.gr/yara-a-beginners-guide.html
https://securityintelligence.com/signature-based-detection-with-yara/
https://countuponsecurity.com/2016/02/10/unleashing-yara-part-1/

YARA signatures:
* https://github.com/Yara-Rules/rules
* http://www.deependresearch.org/2012/08/yara-signature-exchange-google-group.html
* https://malwareconfig.com/yara/

Interesting projects:
* https://github.com/Neo23x0/yarGen
* https://github.com/godaddy/procfilter

NTP synchronization

Upon un-expected power shutdown, my system date/time got changed upon reboot. Here is what I did to correct the situation:

My ntpsync.sh script is as follows:

# Instead of ntpdate (which is deprecated - http://linux.die.net/man/8/ntpd), use
# for older systems, you can use ntpdate like:
# $ sudo service ntp stop
# $ sudo ntpdate -s www.nist.gov
# $ sudo service ntp start

$ sudo service ntp stop
$ sudo ntpd -gq
$ sudo service ntp start

# The -gq tells the ntp daemon to correct the time regardless of the offset (g) and exit immediately (q).

Installation of Kafka on CentOS 7

Apache kafka is open source stream processing platform developed by Apache/LinkedIN and is written in Scala/Java. The project aims to provide a unified, high throughput, low-latency platform for handling real-time data feeds. One of the strongest point of Kafka is massively scalable pub/sub message queue architecture as a distributed transaction log and is suitable for handling streaming data.

It is possible to deploy kafka on a single server or build a distributed kafka cluster for greater performance.

### Update system
$ sudo yum update -y && sudo reboot

### Install OpenJDK runtime
$ sudo yum install java-1.8.0-openjdk.x86_64

Check java version
$ java -version

### Add JAVA_HOME and JRE_HOME in /etc/profile
export JAVA_HOME = /usr/lib/jvm/jre-1.8.0-openjdk
export JRE_HOME = /usr/lib/jvm/jre

Apply the modified profile
$ sudo source /etc/profile

### Download the latest version of Apache Kafka
$ cd ~
$ wget -c https://archive.apache.org/dist/kafka/0.11.0.0/kafka_2.12-0.11.0.0.tgz

Unzip the archive and move to the preferred location such as /opt
$ tar -xvf kafka_2.12-0.11.0.0.tgz
$ sudo mv kafka_2.12-0.11.0.0 /opt

### Start and test Apache Kafka
Go to kafka directory
$ cd /opt/kafka_2.12-0.11.0.0

#### Start Zookeeper server
$ bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

#### Modify configuration of kafka server
$ vim bin/kafka-server-start.sh

Adjust the memory usage according to your specific system parameters.

By default,
export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"

Replace it with:
export KAFKA_HEAP_OPTS="-Xmx512M -Xms256M"

### Start kafka server
$ bin/kafka-server-start.sh config/server.properties

If everything went successfully, you will see several messages about the Kafka server's status, and the last one will read:

INFO [Kafka Server 0], started (kafka.server.KafkaServer)

Congratulations!! you have started kafka server. Press CTRL + C to stop the server.

Now, run kafka in daemon mode like this
$ bin/kafka-server-start.sh -daemon config/server.properties

### Create a topic "test" on Kafka server
$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

If you wish to view the topics, you can view like this:
$ bin/kafka-topics.sh --list --zookeeper localhost:2181

In this case, the output will be:
test

### Produce messages using topic "test"

$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
Now, on command(console) prompt, you can input any number of messages as you wish, such as:
Welcome Joshi
Enjoy Kafka journey!

Uset CTRL + C to stop the messages.

If you receive an error similar to "WARN Error while fetching metadata with correlation id" while inputting a message, you'll need to update the server.properties file with the following info:

port = 9092
advertised.host.name = localhost

### Consume messages
$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

Hola! Whatever you have typed earlier will now be visible on console. Effectively, you have consumed the messages.

### Role of Zookeeper

ZooKeeper coordinates and synchronizes configuration information of distributed nodes. Kafka cluster depends on ZooKeeper to perform operations such as electing leaders and detecting failed nodes.

### Testing zookeeper

Type 'ruok' as telnet console input and the response will be 'imok'

$ telnet localhost 2181
Connected to localhost
Escape character is '^]'.
ruok
imok

### Counting Number of messages stored in a kafka topic
$ bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic test --time -1

This sum up all the counts for each partition.

tldextract python package Error - "ERROR:tldextract:Exception reading Public Suffix List url"

After installing "tldextract" package on a intranet machine for getting domain/subdomain information, I encountered an error:

ERROR:tldextract:Exception reading Public Suffix List url https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat - HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /publicsuffix/list/master/public_suffix_list.dat (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fd627bfd690>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)).
ERROR:tldextract:No Public Suffix List found. Consider using a mirror or constructing your TLDExtract with `suffix_list_urls=None`.

After looking through "tldextract" github repository page on advanced usage (https://github.com/john-kurkowski/tldextract#advanced-usage),
I realized that I have to manually download the public suffix list url for intranet machine during tldextract instance initialization. Basically, you have to construct your own public suffix list manually.

So, I downloaded public suffix list file "public_suffix_list.dat" from url - https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat
and passed on this as an argument to suffix_list_urls.



After setting suffix_list_urls to a file based scheme, it worked without any issue.

Here is the sample script I wrote for my intranet machine testing.

#!/usr/bin/env python
import tldextract
from tldextract.tldextract import LOG
import sys
import logging

# setup logging
logging.basicConfig(stream=sys.stdout,level = logging.DEBUG)
logger = logging.getLogger(__name__)
# If you do not setup logging, you will encounter warning: No handlers could be found for logger "tldextract"


#Download public_suffix_list.dat file from url - https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat

no_fetch_extract = tldextract.TLDExtract(suffix_list_urls=["file:///home/psj/Development/public_suffix_list.dat"],cache_file='/tmp/.tld_set')

print no_fetch_extract('http://www.google.com')
sys.exit(1)

Ref urls:
https://github.com/john-kurkowski/tldextract#advanced-usage
https://github.com/john-kurkowski/tldextract/tree/1.3.1#specifying-your-own-url-or-file-for-the-suffix-list-data

Bro - connection flags

I always forget to remember the conn flags url on Bro site. So, keeping "conn.log" flags state as a reference:

conn_state     Meaning
-----------     --------
S0         Connection attempt seen, no reply.
S1         Connection established, not terminated.
SF         Normal establishment and termination. Note that this is the same symbol as for state S1. You can tell the two apart because for S1 there will not be any byte counts in the summary, while for SF there will be.
REJ         Connection attempt rejected.
S2         Connection established and close attempt by originator seen (but no reply from responder).
S3         Connection established and close attempt by responder seen (but no reply from originator).
RSTO         Connection established, originator aborted (sent a RST).
RSTR         Responder sent a RST.
RSTOS0         Originator sent a SYN followed by a RST, we never saw a SYN-ACK from the responder.
RSTRH         Responder sent a SYN ACK followed by a RST, we never saw a SYN from the (purported) originator.
SH         Originator sent a SYN followed by a FIN, we never saw a SYN ACK from the responder (hence the connection was “half” open).
SHR         Responder sent a SYN ACK followed by a FIN, we never saw a SYN from the originator.
OTH         No SYN seen, just midstream traffic (a “partial connection” that was not later closed).

Ref - https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html

Two-Factor-Authentication with SSH

I wanted to enable two factor authentication for some sensitive servers and realized that openssh supports two factor authentication methods. For this, you require CentOS 7.0 distribution/ Ubuntu 16.10 with OpenSSH server >= 6.6 or more.

My first authentication factor is public-private key based and second factor is password.

Some portion of my sshd configuration is given below:

[root@psj admin]# cat /etc/ssh/sshd_config
HostKey /etc/ssh/ssh_host_rsa_key
SyslogFacility AUTHPRIV
PubkeyAuthentication yes
AuthorizedKeysFile      .ssh/authorized_keys
AuthorizedKeysFile    .ssh/authorized_keys
PasswordAuthentication yes
ChallengeResponseAuthentication no
UsePAM yes
X11Forwarding no
UsePrivilegeSeparation sandbox        # Default for new installations.
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
Subsystem    sftp    /usr/libexec/openssh/sftp-server
Match User admin
    #AuthenticationMethods publickey,password publickey,keyboard-interactive
    AuthenticationMethods publickey,password


If you want to enable "keyboard-interactive" as one of the authentication option, you have to set "ChallengeResponseAuthentication" to yes.


I found the following links to be very useful:

1. https://sysconfig.org.uk/two-factor-authentication-with-ssh.html
2. https://superuser.com/questions/942132/openssh-6-8p1-cant-use-multiple-authenticationmethods
3. https://www.digitalocean.com/community/tutorials/how-to-set-up-multi-factor-authentication-for-ssh-on-ubuntu-16-04

Errors during installation of pycurl

If you are installing pycurl via pip ( a python package manager), you will get a number of errors during the build process. It happens as you are not having the required dependencies installed on the system and these can be fixed easily.

The simple solution to get rid of all the errors mentioned below is (Ubuntu/Debian platform):

$ sudo apt install python-dev libssl-dev libcurl4-openssl-dev

Now, install pycurl as you originally intended!
$ sudo pip install pycurl


The most common errors and their fixes are listed below:

Error - Could not run curl-config: [Errno 2] No such file or directory
--------------------------------------------------------------------------------------------------------------
psj@ubuntu:~/Development$ sudo pip install pycurl
Collecting pycurl
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-hVI9Y_/pycurl/setup.py", line 823, in <module>
        ext = get_extension(sys.argv, split_extension_source=split_extension_source)
      File "/tmp/pip-build-hVI9Y_/pycurl/setup.py", line 497, in get_extension
        ext_config = ExtensionConfiguration(argv)
      File "/tmp/pip-build-hVI9Y_/pycurl/setup.py", line 71, in __init__
        self.configure()
      File "/tmp/pip-build-hVI9Y_/pycurl/setup.py", line 107, in configure_unix
        raise ConfigurationError(msg)
    __main__.ConfigurationError: Could not run curl-config: [Errno 2] No such file or directory

How to fix:
------------------
psj@ubuntu:~/Development$ sudo apt install libcurl4-openssl-dev

Error - openssl/crypto.h: No such file or directory
---------------------------------------------------------------------------------
You may encounter another error:
  In file included from src/docstrings.c:4:0:
    src/pycurl.h:170:31: fatal error: openssl/crypto.h: No such file or directory
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

How to fix:
-------------------
psj@ubuntu:~/Development$ sudo apt install libssl-dev   

Of course, do not forget to install python-dev package.
$ sudo apt install python-dev

Cryptic python subprocess error - OSError: [Errno 2] No such file or directory

Tshark has become swiss army knief for me and I was experimenting with tshark to extract DNS traffic. A portion of the code is reproduced here:

command = /usr/bin/tshark -i ens33 -nn -T fields -e frame.time -e ip.src -e ip.dst -e dns.count.queries -e dns.count.answers -e dns.qry.name -e dns.qry.type -e dns.resp.name -e dns.resp.type -e dns.resp.ttl -e dns.a -e dns.ns -e dns.mx.mail_exchange -e dns.cname -e dns.txt -e dns.flags.rcode -Y 'dns.count.answers gt 0' -E separator='|'

Traceback (most recent call last):
  File "collect.py", line 59, in <module>
    main()
  File "collect.py", line 45, in main
    tshark_response = subprocess.Popen(command, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory


After reading the documentation (http://docs.python.org/2/library/subprocess.html#frequently-used-arguments), I realized that "shell=True" should be added as argument and seems one of the easiest option and Vola! It worked.

Ubuntu Xenial - apt update - E: Some index files failed to download. They have been ignored, or old ones used instead.

Recently, I installed Ubuntu 16.04 on a new PC and updated my repository URLs in "/etc/apt/sources.list" file to point to internal Ubuntu repository servers.


While I was updating my package indexes, I encountered the following errors:

E: Failed to fetch http://repo.xxx.xxx.in/ubuntu/dists/xenial/main/i18n/Translation-en  404  Not Found
E: Failed to fetch http://repo.xxx.xxx.in/ubuntu/dists/xenial-updates/main/i18n/Translation-en  404  Not Found
E: Failed to fetch http://repo.xxx.xxx.in/ubuntu/dists/xenial-backports/main/i18n/Translation-en  404  Not Found
E: Failed to fetch http://repo.xxx.xxx.in/ubuntu/dists/xenial-security/main/i18n/Translation-en  404  Not Found
E: Some index files failed to download. They have been ignored, or old ones used instead.

After searching on google and stackoverflow, it seems a language translation issue. To get rid of this issue, please follow the following steps:

1) Create a file named /etc/apt/apt.conf.d/99translations & add the following content to it:

Acquire::Languages "none";

2) You may also need to remove existing translation files in /var/lib/apt/lists/
$ sudo rm -vf /var/lib/apt/lists/*

3) Now, do cleanup and update of package indexes.

$ sudo apt-get clean
$ sudo apt-get update

Some useful references on stackoverflow:
1) https://askubuntu.com/questions/74653/how-can-i-remove-the-translation-entries-in-apt
2) https://askubuntu.com/questions/762273/16-04-upgrade-failed-to-fetch-empty-files-cant-be-valid-archives/764463

pip based python packages installation through proxy on Windows

For installing python packages through pip via proxy on Windows platform:

1) Create a directory: c:\Users\<User name>\pip
2) Then, create a file "pip.conf" in this directory with the following contents:

[global]
trusted-host = pypi.python.org
proxy = http://user:xxx@192.168.1.4:3128

Now, you are ready to install any python packages through pip:
c:\> pip install requests


Useful references:
1) https://stackoverflow.com/questions/28278207/python-cant-find-pip-ini-or-pip-conf-in-windows7
2) https://stackoverflow.com/questions/14149422/using-pip-behind-a-proxy

Netflow records - TCP flags explained

After setting up nfdump for netflow analysis, I was going through the flow records and got puzzled by flags column. It took some time to understand the concept.


Netflow records contain a field reporting cumulative OR-ed TCP flags seen in the flow. e.g. In a regular TCP connection, a client would sent a SYN, then an ACK, then optional flags like PSH and finally FIN. The table below reports a summary of TCP flags with their binary and decimal values.

Consider two netflow records - a normal TCP connection and closed port

    1) Normal TCP connection - If we sum (using OR operator) all the flags used in a TCP connection (SYN(2) + ACK(16) + PSH(8) + FIN(1)), we have 27.
    Client-to-server: TCP flags = 2 SYN(2)
    Server-to-client: TCP flags = 16 ACK(16)
    Client side Optional flags: TCP flags = 8 PSH(8)

    Client-to-server: TCP flags = 1 FIN(1)

     Cumulative flag value - 2 [SYN(2)] + 16 [ACK(16] + 8 [PSH(8)]+ 1 [FIN(1)]

    2) Closed port - Our client would send a SYN packet while server will reply with RST/ACK packet.So, we have two flows:

    Client-to-server: TCP flags = 2 (SYN(2))
    Server-to-client: TCP flags = 20 (RST(4) + ACK(16))


Good explanation of TCP flags in netflow based on flow-viewer is available here : - https://blog.pierky.com/netflow-weird-tcp-flags-in-flowviewer-and-flow-print/

Elasticsearch - node validation exception bootstrap checks failed on CentOS 6.x

During setting up of elasticsearch 5.x cluster, I encountered "bootstrap checks failed" error as given below:

[2017-02-09T09:47:32,004][ERROR][o.e.b.Bootstrap] [elk5-master-node] node validation exception
bootstrap checks failed
system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2017-02-09T09:47:32,022][INFO ][o.e.n.Node               ] [elk5-master-node] stopping ...
[2017-02-09T09:47:32,163][INFO ][o.e.n.Node               ] [elk5-master-node] stopped
[2017-02-09T09:47:32,163][INFO ][o.e.n.Node               ] [elk5-master-node] closing ...
[2017-02-09T09:47:32,181][INFO ][o.e.n.Node               ] [elk5-master-node] closed


After going through github issue(s), I realized that "secomp" on CentOS 6 is not available and you have to disable it.

There is a discussion on it here -  https://github.com/elastic/elasticsearch/issues/22899

So, add the following settings in elasticsearch.yml:
bootstrap.system_call_filter: false

and start elasticsearch again.

Redis installation error - /deps/hiredis/libhiredis.a: No such file or directory

During compilation of redis from source, I encountered a strange error:

[root@psj redis-3.2.6]# make
cd src && make all
make[1]: Entering directory `/home/joshi/redis-3.2.6/src'
    LINK redis-server
cc: ../deps/hiredis/libhiredis.a: No such file or directory
cc: ../deps/lua/src/liblua.a: No such file or directory
cc: ../deps/jemalloc/lib/libjemalloc.a: No such file or directory
make[1]: *** [redis-server] Error 1
make[1]: Leaving directory `/home/joshi/redis-3.2.6/src'
make: *** [all] Error 2

To get rid of this error, please install the following packages through yum:

yum install gcc gcc-c++ kernel-devel -y
yum install jemalloc jemalloc-devel -y

Logstash error - ERROR StatusLogger No log4j2 configuration file found

I always check logstash configuration file using command line before running logstash service.

After up gradation to Logstash 5, I encountered this error.

ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
2017-01-18 14:29:52,358 main ERROR FileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied)
        at java.io.FileOutputStream.open0(Native Method)
        at java.io.FileOutputStream.open(FileOutputStream.java:270)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
        at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:430)
        at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:403)
        at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:73)
        at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:81)
        at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:103)
        at org.apache.logging.log4j.core.appender.RollingFileAppender.createAppender(RollingFileAppender.java:191)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
 
There were couple of mistakes.
1) I was running logstash on command line as a root. It is required to run it as "logstash" user.
2) I was not giving absolute path for --path.settings and -f options. An interesting discussion thread can be followed here - https://discuss.elastic.co/t/logstash-5-not-running/64449/47

So, if you wish to run logstash(ver 5.x) on command line, you should do this:

Test logstash configurations using command line
----------------------------------------------
[psj@desk joshi]# sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/redis-logstash.conf --config.test_and_exit --verbose


Run logstash on command line and not as a service
-----------------------------------------------------------------------------------
[psj@desk joshi]# sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/redis-logstash.conf

Important notes:

1) Make sure that the options that you give for path.settings and configuration file are absolute.
2) Don't add "/" at the end of directory setting for path.settings (i.e. - /etc/logstash)
3) Keep track of the generated logs under /var/log/logstash/logstash-plain.log

You can change the logging level in log4j2 configuration file by setting status to "info" instead of "error" and you will get to know what is the problem!

 [psj@desk joshi]#head /etc/logstash/log4j2.properties
status = info
name = LogstashPropertiesConfig

Now run logstash on command line - e.g.:

[psj@desk joshi]# sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/redis-logstash.conf

Elasticsearch error - Error - Unable to lock JVM Memory: error=12, reason=Cannot allocate memory

I was doing upgrade to latest version of elasticsearch 5.1 and encountered the following errors:

Error - Unsupported major.minor version 52.0

[root@psj Downloads]# java -version
java version "1.6.0_41"
OpenJDK Runtime Environment (IcedTea6 1.13.13) (rhel-1.13.13.1.el6_8-x86_64)
OpenJDK 64-Bit Server VM (build 23.41-b41, mixed mode)

[root@psj Downloads]# service elasticsearch start
Starting elasticsearch: Exception in thread "main" java.lang.UnsupportedClassVersionError: org/elasticsearch/bootstrap/Elasticsearch : Unsupported major.minor version 52.0
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:648)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:272)
        at java.net.URLClassLoader.access$000(URLClassLoader.java:68)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:207)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:201)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:200)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:325)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:296)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:270)
        at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:406)
                                                           [FAILED]


                                                         
After upgrade of JDK to the most recent version of JDK and re-running of elasticsearch rpm, the above error disappeared!

In another case, I once again encountered the following error:

Error - Unable to lock JVM Memory: error=12, reason=Cannot allocate memory

[2017-01-16T12:17:17,524][WARN ][o.e.b.JNANatives         ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2017-01-16T12:17:17,524][WARN ][o.e.b.JNANatives         ] This can result in part of the JVM being swapped out.
[2017-01-16T12:17:17,525][WARN ][o.e.b.JNANatives         ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2017-01-16T12:17:17,526][WARN ][o.e.b.JNANatives         ] These can be adjusted by modifying /etc/security/limits.conf, for example:
        # allow user 'elasticsearch' mlockall
        elasticsearch soft memlock unlimited
        elasticsearch hard memlock unlimited
[2017-01-16T12:17:17,526][WARN ][o.e.b.JNANatives         ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2017-01-16T12:17:17,868][INFO ][o.e.n.Node               ] [elk5-datanode] initializing ...
[2017-01-16T12:17:18,374][INFO ][o.e.e.NodeEnvironment    ] [elk5-datanode] using [1] data paths, mounts [[/ (/dev/mapper/VolGroup-lv_root)]], net usable_space [3gb], net total_space [6.6gb], spins? [possibly], types [ext4]
[2017-01-16T12:17:18,374][INFO ][o.e.e.NodeEnvironment    ] [elk5-datanode] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-01-16T12:17:18,376][INFO ][o.e.n.Node               ] [elk5-datanode] node name [elk5-datanode], node ID [kXxE48JFRxW_AuiHZr1oJQ]
[2017-01-16T12:17:18,379][INFO ][o.e.n.Node               ] [elk5-datanode] version[5.1.1], pid[20670], build[5395e21/2016-12-06T12:36:15.409Z], OS[Linux/2.6.32-431.1.2.el6.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_71/25.71-b15]
[2017-01-16T12:17:49,880][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [aggs-matrix-stats]
[2017-01-16T12:17:49,880][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [ingest-common]
[2017-01-16T12:17:49,880][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [lang-expression]
[2017-01-16T12:17:49,880][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [lang-groovy]
[2017-01-16T12:17:49,882][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [lang-mustache]
[2017-01-16T12:17:49,882][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [lang-painless]
[2017-01-16T12:17:49,882][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [percolator]
[2017-01-16T12:17:49,882][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [reindex]
[2017-01-16T12:17:49,883][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [transport-netty3]
[2017-01-16T12:17:49,883][INFO ][o.e.p.PluginsService     ] [elk5-datanode] loaded module [transport-netty4]
[2017-01-16T12:17:49,883][INFO ][o.e.p.PluginsService     ] [elk5-datanode] no plugins loaded
[2017-01-16T12:17:57,751][INFO ][o.e.n.Node               ] [elk5-datanode] initialized
[2017-01-16T12:17:57,752][INFO ][o.e.n.Node               ] [elk5-datanode] starting ...
[2017-01-16T12:17:58,179][INFO ][o.e.t.TransportService   ] [elk5-datanode] publish_address {10.35.2.21:9300}, bound_addresses {10.35.2.21:9300}
[2017-01-16T12:17:58,188][INFO ][o.e.b.BootstrapCheck     ] [elk5-datanode] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-01-16T12:17:58,194][ERROR][o.e.b.Bootstrap          ] [elk5-datanode] node validation exception
bootstrap checks failed
memory locking requested for elasticsearch process but memory is not locked
max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]
[2017-01-16T12:17:58,205][INFO ][o.e.n.Node               ] [elk5-datanode] stopping ...
[2017-01-16T12:17:58,253][INFO ][o.e.n.Node               ] [elk5-datanode] stopped
[2017-01-16T12:17:58,253][INFO ][o.e.n.Node               ] [elk5-datanode] closing ...
[2017-01-16T12:17:58,277][INFO ][o.e.n.Node               ] [elk5-datanode] closed


To get rid of this error, add the following contents to "/etc/security/limits.conf" file as shown below:

[root@psj Downloads]# cat /etc/security/limits.conf |grep -v ^#|grep -v ^$
elasticsearch - memlock unlimited
elasticsearch - nproc 4096
* soft - nofile 65536
* hard - nofile 65536
root soft - nofile 65536
root hard - nofile 65536

Then, create a directory:
# mkdir -p /etc/systemd/system/elasticsearch.service.d

# Create a file elasticsearch.conf in this directory with the following content:
[root@psj Downloads]# cat /etc/systemd/system/elasticsearch.service.d/elasticsearch.conf
[Service]
LimitNPROC=2048

If  you have installed pam, pam provides a file /etc/security/limits.d/90-nproc.conf which explicitly overrides the settings in security.conf for the number of processes.

So, modify the file 90-nproc.conf and specify the number of processes:

[root@psj Downloads]# cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     2048
root       soft    nproc     unlimited

Elasticsearch 5 : max number of threads error

I wanted to test the latest version of Elasticsearch released recently. ELK stack has made the log collection super easy and you can quickly generate a nice working system in a jiffy.

So, I downloaded and installed elasticsearch rpm but it failed to start and gave the following error on my CentOS system:

[2017-01-06T10:41:43,737][INFO ][o.e.b.BootstrapCheck     ] [Jeicebz] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-01-06T10:41:43,745][ERROR][o.e.b.Bootstrap          ] [Jeicebz] node validation exception
bootstrap checks failed
max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]
[2017-01-06T10:41:43,749][INFO ][o.e.n.Node               ] [Jeicebz] stopping ...
[2017-01-06T10:41:43,818][INFO ][o.e.n.Node               ] [Jeicebz] stopped
[2017-01-06T10:41:43,819][INFO ][o.e.n.Node               ] [Jeicebz] closing ...
[2017-01-06T10:41:43,838][INFO ][o.e.n.Node               ] [Jeicebz] closed

I tried to setup the security limits as suggested in elastic portal but it was of no help.

As per my belief, my limits.conf (/etc/security/limits.conf) was OK:

[root@psj joshi]# cat /etc/security/limits.conf |grep -v ^#|grep -v ^$
elasticsearch - memlock unlimited
elasticsearch - nproc 4096
* soft - nofile 32768
* hard - nofile 65536
root soft - nofile 32768
root hard - nofile 65536

Finally, the issue turns out to be systemd related idiocracies :

I have to create a file to override the systemd elasticsearch settings as described below:

[root@psj joshi]# mkdir -p /etc/systemd/system/elasticsearch.service.d

[root@psj joshi]# cat /etc/systemd/system/elasticsearch.service.d/elasticsearch.conf
[Service]
LimitNPROC=2048

If  you have installed pam, pam provides a file /etc/security/limits.d/90-nproc.conf which explicitly overrides the settings in security.conf for the number of processes.

So, modify the file 90-nproc.conf and specify the number of processes:


[root@psj joshi]# cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     2048
root       soft    nproc     unlimited

Now, start elasticsearch once again:
# service elasticsearch restart

and Hurry!! it worked!

[root@psj joshi]# tail -f /var/log/elasticsearch/elasticsearch.log
[2017-01-06T13:06:51,672][INFO ][o.e.b.BootstrapCheck     ] [Jeicebz] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-01-06T13:06:54,794][INFO ][o.e.c.s.ClusterService   ] [Jeicebz] new_master {Jeicebz}{JeicebzWSiCZ3kD88HpXbA}{oBmTUCY5QI-IRHx4q5qSCg}{10.3.2.8}{10.3.2.8:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-01-06T13:06:54,836][INFO ][o.e.h.HttpServer         ] [Jeicebz] publish_address {10.35.21.84:9200}, bound_addresses {10.3.2.8:9200}
[2017-01-06T13:06:54,836][INFO ][o.e.n.Node               ] [Jeicebz] started
[2017-01-06T13:06:55,350][INFO ][o.e.g.GatewayService     ] [Jeicebz] recovered [1] indices into cluster_state

The following links were useful:

1. https://underyx.me/2015/05/18/raising-the-maximum-number-of-file-descriptors
2. https://fredrikaverpil.github.io/2016/04/27/systemd-and-resource-limits/ 
3. https://www.elastic.co/guide/en/elasticsearch/reference/master/setting-system-settings.html
4. http://gerardnico.com/wiki/linux/limits.conf