Installation of Kafka on CentOS 7

Apache kafka is open source stream processing platform developed by Apache/LinkedIN and is written in Scala/Java. The project aims to provide a unified, high throughput, low-latency platform for handling real-time data feeds. One of the strongest point of Kafka is massively scalable pub/sub message queue architecture as a distributed transaction log and is suitable for handling streaming data.

It is possible to deploy kafka on a single server or build a distributed kafka cluster for greater performance.

### Update system
$ sudo yum update -y && sudo reboot

### Install OpenJDK runtime
$ sudo yum install java-1.8.0-openjdk.x86_64

Check java version
$ java -version

### Add JAVA_HOME and JRE_HOME in /etc/profile
export JAVA_HOME = /usr/lib/jvm/jre-1.8.0-openjdk
export JRE_HOME = /usr/lib/jvm/jre

Apply the modified profile
$ sudo source /etc/profile

### Download the latest version of Apache Kafka
$ cd ~
$ wget -c https://archive.apache.org/dist/kafka/0.11.0.0/kafka_2.12-0.11.0.0.tgz

Unzip the archive and move to the preferred location such as /opt
$ tar -xvf kafka_2.12-0.11.0.0.tgz
$ sudo mv kafka_2.12-0.11.0.0 /opt

### Start and test Apache Kafka
Go to kafka directory
$ cd /opt/kafka_2.12-0.11.0.0

#### Start Zookeeper server
$ bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

#### Modify configuration of kafka server
$ vim bin/kafka-server-start.sh

Adjust the memory usage according to your specific system parameters.

By default,
export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"

Replace it with:
export KAFKA_HEAP_OPTS="-Xmx512M -Xms256M"

### Start kafka server
$ bin/kafka-server-start.sh config/server.properties

If everything went successfully, you will see several messages about the Kafka server's status, and the last one will read:

INFO [Kafka Server 0], started (kafka.server.KafkaServer)

Congratulations!! you have started kafka server. Press CTRL + C to stop the server.

Now, run kafka in daemon mode like this
$ bin/kafka-server-start.sh -daemon config/server.properties

### Create a topic "test" on Kafka server
$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

If you wish to view the topics, you can view like this:
$ bin/kafka-topics.sh --list --zookeeper localhost:2181

In this case, the output will be:
test

### Produce messages using topic "test"

$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
Now, on command(console) prompt, you can input any number of messages as you wish, such as:
Welcome Joshi
Enjoy Kafka journey!

Uset CTRL + C to stop the messages.

If you receive an error similar to "WARN Error while fetching metadata with correlation id" while inputting a message, you'll need to update the server.properties file with the following info:

port = 9092
advertised.host.name = localhost

### Consume messages
$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

Hola! Whatever you have typed earlier will now be visible on console. Effectively, you have consumed the messages.

### Role of Zookeeper

ZooKeeper coordinates and synchronizes configuration information of distributed nodes. Kafka cluster depends on ZooKeeper to perform operations such as electing leaders and detecting failed nodes.

### Testing zookeeper

Type 'ruok' as telnet console input and the response will be 'imok'

$ telnet localhost 2181
Connected to localhost
Escape character is '^]'.
ruok
imok

### Counting Number of messages stored in a kafka topic
$ bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic test --time -1

This sum up all the counts for each partition.

tldextract python package Error - "ERROR:tldextract:Exception reading Public Suffix List url"

After installing "tldextract" package on a intranet machine for getting domain/subdomain information, I encountered an error:

ERROR:tldextract:Exception reading Public Suffix List url https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat - HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /publicsuffix/list/master/public_suffix_list.dat (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fd627bfd690>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)).
ERROR:tldextract:No Public Suffix List found. Consider using a mirror or constructing your TLDExtract with `suffix_list_urls=None`.

After looking through "tldextract" github repository page on advanced usage (https://github.com/john-kurkowski/tldextract#advanced-usage),
I realized that I have to manually download the public suffix list url for intranet machine during tldextract instance initialization. Basically, you have to construct your own public suffix list manually.

So, I downloaded public suffix list file "public_suffix_list.dat" from url - https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat
and passed on this as an argument to suffix_list_urls.



After setting suffix_list_urls to a file based scheme, it worked without any issue.

Here is the sample script I wrote for my intranet machine testing.

#!/usr/bin/env python
import tldextract
from tldextract.tldextract import LOG
import sys
import logging

# setup logging
logging.basicConfig(stream=sys.stdout,level = logging.DEBUG)
logger = logging.getLogger(__name__)
# If you do not setup logging, you will encounter warning: No handlers could be found for logger "tldextract"


#Download public_suffix_list.dat file from url - https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat

no_fetch_extract = tldextract.TLDExtract(suffix_list_urls=["file:///home/psj/Development/public_suffix_list.dat"],cache_file='/tmp/.tld_set')

print no_fetch_extract('http://www.google.com')
sys.exit(1)

Ref urls:
https://github.com/john-kurkowski/tldextract#advanced-usage
https://github.com/john-kurkowski/tldextract/tree/1.3.1#specifying-your-own-url-or-file-for-the-suffix-list-data

Bro - connection flags

I always forget to remember the conn flags url on Bro site. So, keeping "conn.log" flags state as a reference:

conn_state     Meaning
-----------     --------
S0         Connection attempt seen, no reply.
S1         Connection established, not terminated.
SF         Normal establishment and termination. Note that this is the same symbol as for state S1. You can tell the two apart because for S1 there will not be any byte counts in the summary, while for SF there will be.
REJ         Connection attempt rejected.
S2         Connection established and close attempt by originator seen (but no reply from responder).
S3         Connection established and close attempt by responder seen (but no reply from originator).
RSTO         Connection established, originator aborted (sent a RST).
RSTR         Responder sent a RST.
RSTOS0         Originator sent a SYN followed by a RST, we never saw a SYN-ACK from the responder.
RSTRH         Responder sent a SYN ACK followed by a RST, we never saw a SYN from the (purported) originator.
SH         Originator sent a SYN followed by a FIN, we never saw a SYN ACK from the responder (hence the connection was “half” open).
SHR         Responder sent a SYN ACK followed by a FIN, we never saw a SYN from the originator.
OTH         No SYN seen, just midstream traffic (a “partial connection” that was not later closed).

Ref - https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html

Two-Factor-Authentication with SSH

I wanted to enable two factor authentication for some sensitive servers and realized that openssh supports two factor authentication methods. For this, you require CentOS 7.0 distribution/ Ubuntu 16.10 with OpenSSH server >= 6.6 or more.

My first authentication factor is public-private key based and second factor is password.

Some portion of my sshd configuration is given below:

[root@psj admin]# cat /etc/ssh/sshd_config
HostKey /etc/ssh/ssh_host_rsa_key
SyslogFacility AUTHPRIV
PubkeyAuthentication yes
AuthorizedKeysFile      .ssh/authorized_keys
AuthorizedKeysFile    .ssh/authorized_keys
PasswordAuthentication yes
ChallengeResponseAuthentication no
UsePAM yes
X11Forwarding no
UsePrivilegeSeparation sandbox        # Default for new installations.
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
Subsystem    sftp    /usr/libexec/openssh/sftp-server
Match User admin
    #AuthenticationMethods publickey,password publickey,keyboard-interactive
    AuthenticationMethods publickey,password


If you want to enable "keyboard-interactive" as one of the authentication option, you have to set "ChallengeResponseAuthentication" to yes.


I found the following links to be very useful:

1. https://sysconfig.org.uk/two-factor-authentication-with-ssh.html
2. https://superuser.com/questions/942132/openssh-6-8p1-cant-use-multiple-authenticationmethods
3. https://www.digitalocean.com/community/tutorials/how-to-set-up-multi-factor-authentication-for-ssh-on-ubuntu-16-04

Errors during installation of pycurl

If you are installing pycurl via pip ( a python package manager), you will get a number of errors during the build process. It happens as you are not having the required dependencies installed on the system and these can be fixed easily.

The simple solution to get rid of all the errors mentioned below is (Ubuntu/Debian platform):

$ sudo apt install python-dev libssl-dev libcurl4-openssl-dev

Now, install pycurl as you originally intended!
$ sudo pip install pycurl


The most common errors and their fixes are listed below:

Error - Could not run curl-config: [Errno 2] No such file or directory
--------------------------------------------------------------------------------------------------------------
psj@ubuntu:~/Development$ sudo pip install pycurl
Collecting pycurl
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-hVI9Y_/pycurl/setup.py", line 823, in <module>
        ext = get_extension(sys.argv, split_extension_source=split_extension_source)
      File "/tmp/pip-build-hVI9Y_/pycurl/setup.py", line 497, in get_extension
        ext_config = ExtensionConfiguration(argv)
      File "/tmp/pip-build-hVI9Y_/pycurl/setup.py", line 71, in __init__
        self.configure()
      File "/tmp/pip-build-hVI9Y_/pycurl/setup.py", line 107, in configure_unix
        raise ConfigurationError(msg)
    __main__.ConfigurationError: Could not run curl-config: [Errno 2] No such file or directory

How to fix:
------------------
psj@ubuntu:~/Development$ sudo apt install libcurl4-openssl-dev

Error - openssl/crypto.h: No such file or directory
---------------------------------------------------------------------------------
You may encounter another error:
  In file included from src/docstrings.c:4:0:
    src/pycurl.h:170:31: fatal error: openssl/crypto.h: No such file or directory
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

How to fix:
-------------------
psj@ubuntu:~/Development$ sudo apt install libssl-dev   

Of course, do not forget to install python-dev package.
$ sudo apt install python-dev

Cryptic python subprocess error - OSError: [Errno 2] No such file or directory

Tshark has become swiss army knief for me and I was experimenting with tshark to extract DNS traffic. A portion of the code is reproduced here:

command = /usr/bin/tshark -i ens33 -nn -T fields -e frame.time -e ip.src -e ip.dst -e dns.count.queries -e dns.count.answers -e dns.qry.name -e dns.qry.type -e dns.resp.name -e dns.resp.type -e dns.resp.ttl -e dns.a -e dns.ns -e dns.mx.mail_exchange -e dns.cname -e dns.txt -e dns.flags.rcode -Y 'dns.count.answers gt 0' -E separator='|'

Traceback (most recent call last):
  File "collect.py", line 59, in <module>
    main()
  File "collect.py", line 45, in main
    tshark_response = subprocess.Popen(command, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory


After reading the documentation (http://docs.python.org/2/library/subprocess.html#frequently-used-arguments), I realized that "shell=True" should be added as argument and seems one of the easiest option and Vola! It worked.

Ubuntu Xenial - apt update - E: Some index files failed to download. They have been ignored, or old ones used instead.

Recently, I installed Ubuntu 16.04 on a new PC and updated my repository URLs in "/etc/apt/sources.list" file to point to internal Ubuntu repository servers.


While I was updating my package indexes, I encountered the following errors:

E: Failed to fetch http://repo.xxx.xxx.in/ubuntu/dists/xenial/main/i18n/Translation-en  404  Not Found
E: Failed to fetch http://repo.xxx.xxx.in/ubuntu/dists/xenial-updates/main/i18n/Translation-en  404  Not Found
E: Failed to fetch http://repo.xxx.xxx.in/ubuntu/dists/xenial-backports/main/i18n/Translation-en  404  Not Found
E: Failed to fetch http://repo.xxx.xxx.in/ubuntu/dists/xenial-security/main/i18n/Translation-en  404  Not Found
E: Some index files failed to download. They have been ignored, or old ones used instead.

After searching on google and stackoverflow, it seems a language translation issue. To get rid of this issue, please follow the following steps:

1) Create a file named /etc/apt/apt.conf.d/99translations & add the following content to it:

Acquire::Languages "none";

2) You may also need to remove existing translation files in /var/lib/apt/lists/
$ sudo rm -vf /var/lib/apt/lists/*

3) Now, do cleanup and update of package indexes.

$ sudo apt-get clean
$ sudo apt-get update

Some useful references on stackoverflow:
1) https://askubuntu.com/questions/74653/how-can-i-remove-the-translation-entries-in-apt
2) https://askubuntu.com/questions/762273/16-04-upgrade-failed-to-fetch-empty-files-cant-be-valid-archives/764463