Two-Factor-Authentication with SSH

I wanted to enable two factor authentication for some sensitive servers and realized that openssh supports two factor authentication methods. For this, you require CentOS 7.0 distribution/ Ubuntu 16.10 with OpenSSH server >= 6.6 or more.

My first authentication factor is public-private key based and second factor is password.

Some portion of my sshd configuration is given below:

[root@psj admin]# cat /etc/ssh/sshd_config
HostKey /etc/ssh/ssh_host_rsa_key
SyslogFacility AUTHPRIV
PubkeyAuthentication yes
AuthorizedKeysFile      .ssh/authorized_keys
AuthorizedKeysFile    .ssh/authorized_keys
PasswordAuthentication yes
ChallengeResponseAuthentication no
UsePAM yes
X11Forwarding no
UsePrivilegeSeparation sandbox        # Default for new installations.
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
Subsystem    sftp    /usr/libexec/openssh/sftp-server
Match User admin
    #AuthenticationMethods publickey,password publickey,keyboard-interactive
    AuthenticationMethods publickey,password


If you want to enable "keyboard-interactive" as one of the authentication option, you have to set "ChallengeResponseAuthentication" to yes.


I found the following links to be very useful:

1. https://sysconfig.org.uk/two-factor-authentication-with-ssh.html
2. https://superuser.com/questions/942132/openssh-6-8p1-cant-use-multiple-authenticationmethods
3. https://www.digitalocean.com/community/tutorials/how-to-set-up-multi-factor-authentication-for-ssh-on-ubuntu-16-04

Errors during installation of pycurl

If you are installing pycurl via pip ( a python package manager), you will get a number of errors during the build process. It happens as you are not having the required dependencies installed on the system and these can be fixed easily.

The simple solution to get rid of all the errors mentioned below is (Ubuntu/Debian platform):

$ sudo apt install python-dev libssl-dev libcurl4-openssl-dev

Now, install pycurl as you originally intended!
$ sudo pip install pycurl


The most common errors and their fixes are listed below:

Error - Could not run curl-config: [Errno 2] No such file or directory
--------------------------------------------------------------------------------------------------------------
psj@ubuntu:~/Development$ sudo pip install pycurl
Collecting pycurl
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-hVI9Y_/pycurl/setup.py", line 823, in <module>
        ext = get_extension(sys.argv, split_extension_source=split_extension_source)
      File "/tmp/pip-build-hVI9Y_/pycurl/setup.py", line 497, in get_extension
        ext_config = ExtensionConfiguration(argv)
      File "/tmp/pip-build-hVI9Y_/pycurl/setup.py", line 71, in __init__
        self.configure()
      File "/tmp/pip-build-hVI9Y_/pycurl/setup.py", line 107, in configure_unix
        raise ConfigurationError(msg)
    __main__.ConfigurationError: Could not run curl-config: [Errno 2] No such file or directory

How to fix:
------------------
psj@ubuntu:~/Development$ sudo apt install libcurl4-openssl-dev

Error - openssl/crypto.h: No such file or directory
---------------------------------------------------------------------------------
You may encounter another error:
  In file included from src/docstrings.c:4:0:
    src/pycurl.h:170:31: fatal error: openssl/crypto.h: No such file or directory
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

How to fix:
-------------------
psj@ubuntu:~/Development$ sudo apt install libssl-dev   

Of course, do not forget to install python-dev package.
$ sudo apt install python-dev

Cryptic python subprocess error - OSError: [Errno 2] No such file or directory

Tshark has become swiss army knief for me and I was experimenting with tshark to extract DNS traffic. A portion of the code is reproduced here:

command = /usr/bin/tshark -i ens33 -nn -T fields -e frame.time -e ip.src -e ip.dst -e dns.count.queries -e dns.count.answers -e dns.qry.name -e dns.qry.type -e dns.resp.name -e dns.resp.type -e dns.resp.ttl -e dns.a -e dns.ns -e dns.mx.mail_exchange -e dns.cname -e dns.txt -e dns.flags.rcode -Y 'dns.count.answers gt 0' -E separator='|'

Traceback (most recent call last):
  File "collect.py", line 59, in <module>
    main()
  File "collect.py", line 45, in main
    tshark_response = subprocess.Popen(command, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory


After reading the documentation (http://docs.python.org/2/library/subprocess.html#frequently-used-arguments), I realized that "shell=True" should be added as argument and seems one of the easiest option and Vola! It worked.

Ubuntu Xenial - apt update - E: Some index files failed to download. They have been ignored, or old ones used instead.

Recently, I installed Ubuntu 16.04 on a new PC and updated my repository URLs in "/etc/apt/sources.list" file to point to internal Ubuntu repository servers.


While I was updating my package indexes, I encountered the following errors:

E: Failed to fetch http://repo.xxx.xxx.in/ubuntu/dists/xenial/main/i18n/Translation-en  404  Not Found
E: Failed to fetch http://repo.xxx.xxx.in/ubuntu/dists/xenial-updates/main/i18n/Translation-en  404  Not Found
E: Failed to fetch http://repo.xxx.xxx.in/ubuntu/dists/xenial-backports/main/i18n/Translation-en  404  Not Found
E: Failed to fetch http://repo.xxx.xxx.in/ubuntu/dists/xenial-security/main/i18n/Translation-en  404  Not Found
E: Some index files failed to download. They have been ignored, or old ones used instead.

After searching on google and stackoverflow, it seems a language translation issue. To get rid of this issue, please follow the following steps:

1) Create a file named /etc/apt/apt.conf.d/99translations & add the following content to it:

Acquire::Languages "none";

2) You may also need to remove existing translation files in /var/lib/apt/lists/
$ sudo rm -vf /var/lib/apt/lists/*

3) Now, do cleanup and update of package indexes.

$ sudo apt-get clean
$ sudo apt-get update

Some useful references on stackoverflow:
1) https://askubuntu.com/questions/74653/how-can-i-remove-the-translation-entries-in-apt
2) https://askubuntu.com/questions/762273/16-04-upgrade-failed-to-fetch-empty-files-cant-be-valid-archives/764463

pip based python packages installation through proxy on Windows

For installing python packages through pip via proxy on Windows platform:

1) Create a directory: c:\Users\<User name>\pip
2) Then, create a file "pip.conf" in this directory with the following contents:

[global]
trusted-host = pypi.python.org
proxy = http://user:xxx@192.168.1.4:3128

Now, you are ready to install any python packages through pip:
c:\> pip install requests


Useful references:
1) https://stackoverflow.com/questions/28278207/python-cant-find-pip-ini-or-pip-conf-in-windows7
2) https://stackoverflow.com/questions/14149422/using-pip-behind-a-proxy

Netflow records - TCP flags explained

After setting up nfdump for netflow analysis, I was going through the flow records and got puzzled by flags column. It took some time to understand the concept.


Netflow records contain a field reporting cumulative OR-ed TCP flags seen in the flow. e.g. In a regular TCP connection, a client would sent a SYN, then an ACK, then optional flags like PSH and finally FIN. The table below reports a summary of TCP flags with their binary and decimal values.

Consider two netflow records - a normal TCP connection and closed port

    1) Normal TCP connection - If we sum (using OR operator) all the flags used in a TCP connection (SYN(2) + ACK(16) + PSH(8) + FIN(1)), we have 27.
    Client-to-server: TCP flags = 2 SYN(2)
    Server-to-client: TCP flags = 16 ACK(16)
    Client side Optional flags: TCP flags = 8 PSH(8)

    Client-to-server: TCP flags = 1 FIN(1)

     Cumulative flag value - 2 [SYN(2)] + 16 [ACK(16] + 8 [PSH(8)]+ 1 [FIN(1)]

    2) Closed port - Our client would send a SYN packet while server will reply with RST/ACK packet.So, we have two flows:

    Client-to-server: TCP flags = 2 (SYN(2))
    Server-to-client: TCP flags = 20 (RST(4) + ACK(16))


Good explanation of TCP flags in netflow based on flow-viewer is available here : - https://blog.pierky.com/netflow-weird-tcp-flags-in-flowviewer-and-flow-print/

Elasticsearch - node validation exception bootstrap checks failed on CentOS 6.x

During setting up of elasticsearch 5.x cluster, I encountered "bootstrap checks failed" error as given below:

[2017-02-09T09:47:32,004][ERROR][o.e.b.Bootstrap] [elk5-master-node] node validation exception
bootstrap checks failed
system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2017-02-09T09:47:32,022][INFO ][o.e.n.Node               ] [elk5-master-node] stopping ...
[2017-02-09T09:47:32,163][INFO ][o.e.n.Node               ] [elk5-master-node] stopped
[2017-02-09T09:47:32,163][INFO ][o.e.n.Node               ] [elk5-master-node] closing ...
[2017-02-09T09:47:32,181][INFO ][o.e.n.Node               ] [elk5-master-node] closed


After going through github issue(s), I realized that "secomp" on CentOS 6 is not available and you have to disable it.

There is a discussion on it here -  https://github.com/elastic/elasticsearch/issues/22899

So, add the following settings in elasticsearch.yml:
bootstrap.system_call_filter: false

and start elasticsearch again.