Download gz file using python request module

Here is the quick script I wrote to download a gz file using python requests module:

#!/usr/bin/env python
import requests
import gzip
import logging
import sys
import StringIO
import zlib

# setup logging
logging.basicConfig(stream = sys.stdout, level = logging.ERROR)
log = logging.getLogger('threat-feeds-logger')

#proxy configuration
proxy_host='10.1.1.11'
proxy_port=3128
proxy_user = 'xxxx'
proxy_password = 'xxxx'

feed_url = 'https://foo.org/foo.gz'
proxy_dict = {
                'http':'http://%s:%s@%s:%s' % (proxy_user, proxy_password, proxy_host, proxy_port),
                'https':'http://%s:%s@%s:%s' % (proxy_user, proxy_password, proxy_host, proxy_port)
            }
try:
    response = requests.get(feed_url,proxies = proxy_dict)
except Exception as e:
    log.error("Error while getting data from url - %s" %feed_url)

if response.status_code == 200:
    buf_data = StringIO.StringIO(response.content)
    f = gzip.GzipFile(fileobj=buf_data)
    for row in f.readlines():
       print row
 

How to find google IP address range

I wanted to track google IP range in one of the investigation. So, I did this:

[admin@psj ~]$ nslookup -type=txt _spf.google.com 8.8.8.8
Server:        8.8.8.8
Address:    8.8.8.8#53

Non-authoritative answer:
_spf.google.com    text = "v=spf1 include:_netblocks.google.com include:_netblocks2.google.com include:_netblocks3.google.com ~all"

Authoritative answers can be found from:
google.com    nameserver = ns2.google.com.
google.com    nameserver = ns3.google.com.
google.com    nameserver = ns1.google.com.
google.com    nameserver = ns4.google.com.
ns2.google.com    internet address = 216.239.34.10
ns3.google.com    internet address = 216.239.36.10
ns1.google.com    internet address = 216.239.32.10
ns4.google.com    internet address = 216.239.38.10

Run a nslookup for each one:
$ nslookup -q=TXT _netblocks.google.com 8.8.8.8
$ nslookup -q=TXT _netblocks2.google.com 8.8.8.8


[admin@psj ~]$ nslookup -type=txt _netblocks.google.com
Server:        8.8.8.8
Address:    8.8.8.8#53

Non-authoritative answer:
_netblocks.google.com    text = "v=spf1 ip4:64.18.0.0/20 ip4:64.233.160.0/19 ip4:66.102.0.0/20 ip4:66.249.80.0/20 ip4:72.14.192.0/18 ip4:74.125.0.0/16 ip4:108.177.8.0/21 ip4:173.194.0.0/16 ip4:207.126.144.0/20 ip4:209.85.128.0/17 ip4:216.58.192.0/19 ip4:216.239.32.0/19 ~all"

Authoritative answers can be found from:
google.com    nameserver = ns2.google.com.
google.com    nameserver = ns3.google.com.
google.com    nameserver = ns4.google.com.
google.com    nameserver = ns1.google.com.
ns2.google.com    internet address = 216.239.34.10
ns3.google.com    internet address = 216.239.36.10
ns1.google.com    internet address = 216.239.32.10
ns4.google.com    internet address = 216.239.38.10

You can also use whois query to find out network blocks assigned to google.
$ whois 74.125.127.108

Using Bro Threat intelligence framework

** Download a malicious pcap from internet, say, sample1.pcap

** Ensure that you have installed bro correctly and bro intelligence frameworks is present with required modules(Minimum version of bro - 2.4 or greater)

[root@ joshi]# ls -l /usr/local/bro/share/bro/policy/frameworks/intel/
do_notice.bro  seen/
If you want to see if intelligence framework scripts are loaded or not at runtime, kindly look in loaded_scripts.log

** Create a intelligence feed file manually, say, intel1.dat:
(Please note that the separator is TAB and not spaces for columns in the file.)

[root@ joshi]# cat intel1.txt
#fields indicator       indicator_type  meta.source     meta.url        meta.do_notice
172.16.88.10    Intel::ADDR     ciarmy  http://www.ciarmy.com/list/ci-badguys.txt       T


** Create a file: intel.bro in say - /home/joshi with the following contents:

# intelligence framework
@load frameworks/intel/seen
@load frameworks/intel/do_notice

redef Intel::read_files += {
    "/home/joshi/intel1.txt"
};

Another variation:
[root@ joshi]# cat intel1.bro
# intelligence framework
@load frameworks/intel/seen
@load frameworks/intel/do_notice

redef Intel::read_files += {
#    "/home/joshi/intel1.txt"
 @DIR + "/intel1.txt",
};

Now, run bro to make use of "intel1.bro" for finding any malicious connections in the pcap.

[root@ joshi]# bro -C -r /home/joshi/Downloads/sample1.pcap intel1.bro


If you wish, you can also add the above lines to site policy script(local.bro) under /usr/local/bro/share/bro/site/local.bro instead of intel1.bro

By default, the logs will be created in the current directory.

To run local script based on site policy(local.bro):
[root@ joshi]# bro -C -r /home/joshi/Downloads/sample1.pcap local

[root@ joshi]# ls -l *.log
-rw-r--r-- 1 root root 60758 Oct  5 17:38 conn.log
-rw-r--r-- 1 root root     0 Oct  5 17:38 debug.log
-rw-r--r-- 1 root root 25454 Oct  5 17:38 dns.log
-rw-r--r-- 1 root root  3736 Oct  5 17:38 http.log
-rw-r--r-- 1 root root  1547 Oct  5 17:38 intel.log
-rw-r--r-- 1 root root 23896 Oct  5 17:36 loaded_scripts.log
-rw-r--r-- 1 root root  2690 Oct  5 17:38 notice.log
-rw-r--r-- 1 root root   253 Oct  5 17:38 packet_filter.log
-rw-r--r-- 1 root root   384 Oct  5 17:38 weird.log

[root@ joshi]# head intel.log
#separator \x09
#set_separator  ,
#empty_field    (empty)
#unset_field    -
#path   intel
#open   2017-10-05-17-38-26
#fields ts      uid     id.orig_h       id.orig_p       id.resp_h       id.resp_p       fuid file_mime_type   file_desc       seen.indicator  seen.indicator_type     seen.where      seen.node     sources
#types  time    string  addr    port    addr    port    string  string  string  string  enum enum     string  set[string]
1394196008.595180       CQRmeTHpuDAbRlDrf       172.16.88.10    49493   172.16.88.135   80   --       -       172.16.88.135   Intel::ADDR     Conn::IN_RESP   bro     ciarmy
1394196043.661031       Ch4sqx4DcNd8kxx5hj      172.16.88.10    49495   172.16.88.135   80   --       -       172.16.88.135   Intel::ADDR     Conn::IN_RESP   bro     ciarmy


Local networks on command line:
[root@ joshi]# bro -C -r /home/joshi/Downloads/sample1.pcap local "Site::local_nets += {10.0.0.0/8,192.168.0.0/16}"

Note: "-C" command line flag for bro is a MUST. I have to spend few hours to understand the behaviour and finally, it was discovered in bro faq(http://www.bro.org/documentation/faq.html)

It allows bro's event engine to process the packets event if packets don't have valid checksums.

The following links are useful if you want additional information:
*  Bro Intel framework - https://www.bro.org/sphinx-git/frameworks/intel.html
*  PCAP files - https://github.com/aboutsecurity/Bro-samples/
*  http://blog.bro.org/2014/01/intelligence-data-and-bro_4980.html

Installing yara

Installation of yara
-----------------------
$ wget https://github.com/VirusTotal/yara/archive/v3.5.0.tar.gz
$ cd yara-3.5.0
$ ./bootstrap.sh
$ ./configure
$ make
# to spot any errors
$ make check
# make install
yara documentation is very good and you can find installation instructions on variety of
platforms. Please refer - http://yara.readthedocs.io/en/latest/gettingstarted.html#

Test yara with your own rule!
$ echo "rule dummy { condition: true }" > my_rule
$ yara -r my_rule my_rule
dummy my_rule

Yara links
----------
Yara repository - https://github.com/VirusTotal/yara/archive/v3.5.0.tar.gz
yara rules - https://github.com/Yara-Rules/rules
https://bruteforce.gr/yara-a-beginners-guide.html
https://securityintelligence.com/signature-based-detection-with-yara/
https://countuponsecurity.com/2016/02/10/unleashing-yara-part-1/

YARA signatures:
* https://github.com/Yara-Rules/rules
* http://www.deependresearch.org/2012/08/yara-signature-exchange-google-group.html
* https://malwareconfig.com/yara/

Interesting projects:
* https://github.com/Neo23x0/yarGen
* https://github.com/godaddy/procfilter

NTP synchronization

Upon un-expected power shutdown, my system date/time got changed upon reboot. Here is what I did to correct the situation:

My ntpsync.sh script is as follows:

# Instead of ntpdate (which is deprecated - http://linux.die.net/man/8/ntpd), use
# for older systems, you can use ntpdate like:
# $ sudo service ntp stop
# $ sudo ntpdate -s www.nist.gov
# $ sudo service ntp start

$ sudo service ntp stop
$ sudo ntpd -gq
$ sudo service ntp start

# The -gq tells the ntp daemon to correct the time regardless of the offset (g) and exit immediately (q).

Installation of Kafka on CentOS 7

Apache kafka is open source stream processing platform developed by Apache/LinkedIN and is written in Scala/Java. The project aims to provide a unified, high throughput, low-latency platform for handling real-time data feeds. One of the strongest point of Kafka is massively scalable pub/sub message queue architecture as a distributed transaction log and is suitable for handling streaming data.

It is possible to deploy kafka on a single server or build a distributed kafka cluster for greater performance.

### Update system
$ sudo yum update -y && sudo reboot

### Install OpenJDK runtime
$ sudo yum install java-1.8.0-openjdk.x86_64

Check java version
$ java -version

### Add JAVA_HOME and JRE_HOME in /etc/profile
export JAVA_HOME = /usr/lib/jvm/jre-1.8.0-openjdk
export JRE_HOME = /usr/lib/jvm/jre

Apply the modified profile
$ sudo source /etc/profile

### Download the latest version of Apache Kafka
$ cd ~
$ wget -c https://archive.apache.org/dist/kafka/0.11.0.0/kafka_2.12-0.11.0.0.tgz

Unzip the archive and move to the preferred location such as /opt
$ tar -xvf kafka_2.12-0.11.0.0.tgz
$ sudo mv kafka_2.12-0.11.0.0 /opt

### Start and test Apache Kafka
Go to kafka directory
$ cd /opt/kafka_2.12-0.11.0.0

#### Start Zookeeper server
$ bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

#### Modify configuration of kafka server
$ vim bin/kafka-server-start.sh

Adjust the memory usage according to your specific system parameters.

By default,
export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"

Replace it with:
export KAFKA_HEAP_OPTS="-Xmx512M -Xms256M"

### Start kafka server
$ bin/kafka-server-start.sh config/server.properties

If everything went successfully, you will see several messages about the Kafka server's status, and the last one will read:

INFO [Kafka Server 0], started (kafka.server.KafkaServer)

Congratulations!! you have started kafka server. Press CTRL + C to stop the server.

Now, run kafka in daemon mode like this
$ bin/kafka-server-start.sh -daemon config/server.properties

### Create a topic "test" on Kafka server
$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

If you wish to view the topics, you can view like this:
$ bin/kafka-topics.sh --list --zookeeper localhost:2181

In this case, the output will be:
test

### Produce messages using topic "test"

$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
Now, on command(console) prompt, you can input any number of messages as you wish, such as:
Welcome Joshi
Enjoy Kafka journey!

Uset CTRL + C to stop the messages.

If you receive an error similar to "WARN Error while fetching metadata with correlation id" while inputting a message, you'll need to update the server.properties file with the following info:

port = 9092
advertised.host.name = localhost

### Consume messages
$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

Hola! Whatever you have typed earlier will now be visible on console. Effectively, you have consumed the messages.

### Role of Zookeeper

ZooKeeper coordinates and synchronizes configuration information of distributed nodes. Kafka cluster depends on ZooKeeper to perform operations such as electing leaders and detecting failed nodes.

### Testing zookeeper

Type 'ruok' as telnet console input and the response will be 'imok'

$ telnet localhost 2181
Connected to localhost
Escape character is '^]'.
ruok
imok

### Counting Number of messages stored in a kafka topic
$ bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic test --time -1

This sum up all the counts for each partition.

tldextract python package Error - "ERROR:tldextract:Exception reading Public Suffix List url"

After installing "tldextract" package on a intranet machine for getting domain/subdomain information, I encountered an error:

ERROR:tldextract:Exception reading Public Suffix List url https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat - HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /publicsuffix/list/master/public_suffix_list.dat (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fd627bfd690>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)).
ERROR:tldextract:No Public Suffix List found. Consider using a mirror or constructing your TLDExtract with `suffix_list_urls=None`.

After looking through "tldextract" github repository page on advanced usage (https://github.com/john-kurkowski/tldextract#advanced-usage),
I realized that I have to manually download the public suffix list url for intranet machine during tldextract instance initialization. Basically, you have to construct your own public suffix list manually.

So, I downloaded public suffix list file "public_suffix_list.dat" from url - https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat
and passed on this as an argument to suffix_list_urls.



After setting suffix_list_urls to a file based scheme, it worked without any issue.

Here is the sample script I wrote for my intranet machine testing.

#!/usr/bin/env python
import tldextract
from tldextract.tldextract import LOG
import sys
import logging

# setup logging
logging.basicConfig(stream=sys.stdout,level = logging.DEBUG)
logger = logging.getLogger(__name__)
# If you do not setup logging, you will encounter warning: No handlers could be found for logger "tldextract"


#Download public_suffix_list.dat file from url - https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat

no_fetch_extract = tldextract.TLDExtract(suffix_list_urls=["file:///home/psj/Development/public_suffix_list.dat"],cache_file='/tmp/.tld_set')

print no_fetch_extract('http://www.google.com')
sys.exit(1)

Ref urls:
https://github.com/john-kurkowski/tldextract#advanced-usage
https://github.com/john-kurkowski/tldextract/tree/1.3.1#specifying-your-own-url-or-file-for-the-suffix-list-data