Monday, 14 September 2015

InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail.

While retriving C&C servers for palevo bots, I encountered an error -

 "InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail....."

psj@psj-desktop:~/Developement/palevo$ python requests_palevo_domains.py
/usr/local/lib/python2.6/dist-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
Traceback (most recent call last):
  File "requests_proxy_usage.py", line 9, in <module>
    r=requests.get('https://palevotracker.abuse.ch/blocklists.php?download=domainblocklist',proxies=proxy_dict)
  File "/usr/local/lib/python2.6/dist-packages/requests/api.py", line 69, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python2.6/dist-packages/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.6/dist-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.6/dist-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.6/dist-packages/requests/adapters.py", line 431, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno 8] _ssl.c:480: EOF occurred in violation of protocol

There are multiple ways to overcome this issue:

1) Upgrade to python 2.7.9  as suggested in urllib3 documentation - https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning

2) By default, python standard library’s ssl module is used. Unfortunately, there are several limitations which are addressed by PyOpenSSL:
  • (Python 2.x) SNI support.
  • (Python 2.x-3.2) Disabling compression to mitigate CRIME attack.
To use the Python OpenSSL bindings instead, you’ll need to install the required packages:
 
$ pip install --upgrade pyopenssl ndg-httpsclient pyasn1
 
or 
 
$ pip install requests[security] 

Thursday, 10 September 2015

Keeping track of programs generating TCP/UDP traffic on Windows

While doing routine security investigation, there was a requirement to track the program generating some TCP traffic. I made use of Sysinternal's TCPView to find out the offending program.

Here are some other options, if you are interested.

1) TCPView - It is a Windows program that will show you detailed listings of all TCP and UDP endpoints on your system, including the local and remote addresses and state of TCP connections.

Download link - https://technet.microsoft.com/en-us/library/bb897437.aspx

2)Microsoft network monitor is another tool.

Download link - https://www.microsoft.com/en-us/download/details.aspx?id=4865

If interested in commericial version, NetBalancer is a good utility.

Download link -
  1. Netbalancer - https://netbalancer.com/download
  2. Command line version - https://netbalancer.com/docs#command_line__nbcmd_exe_

If you have installed Sysinternal utilities on PC, there is a utility namely, Tcpvcon.
Tcpvcon usage is similar to that of the built-in Windows netstat utility.

Usage: tcpvcon [-a] [-c] [-n] [process name or PID]
-a     Show all endpoints (default is to show established TCP connections).
-c     Print output as CSV.
-n     Don't resolve addresses..

If you wish to stick to old netstat utility, here is the way to find out the program ID making TCP/UDP connections:

c:\> netstat -nab

Tuesday, 21 July 2015

Securing Windows/Linux machines using OVAL

Keeping windows/linux machines safe and secure is always a daunting task. The problem is further compounded if you have to deal with multiple windows/linux machines with a wide variety of versions (right from Windows XP-SP3 to Window 8, and Linux variants like CentOS,Scientific Linux, RedHat etc). The end-user is careless most of the time and it is duty of system administrators to remind him that some of the settings are not OK from the security point-of-view. Sometimes, users tweak in windows settings for operational reasons. So, it's necessary to do timely system checks and take corrective actions. Since the manual process is always time consuming and error prone, I was looking for vulnerability compliance solution which is free. Of course, there are number of powerful commercial tools for enforcing Windows policy checks from McAfee,TripWire etc and if you can afford them, go for them!!

Because of budget constraints, I decided to stick to free tool - OVAL. Though the tool does not have a polished interface unlike the commercial counterparts, it does a decent job of finding the security state of system. One big advantage is that Ovaldi is cross-platform and can run on both Windows as well as Linux. Running OVAL scans allowed me to automate the scans of many Windows and Linux system and it is possible to achieve consistency and accuracy across different machines.

For detailed information about OVAL, please visit - http://oval.mitre.org

OVAL is a language that describes checks to be made. These checks are usually conditional i.e. whether a particular audit setting exists or not , or if a particular component is installed or not. Further, they can be grouped with operators like AND, OR and NOT.

If you wish to download complete oval database, please visit - http://oval.mitre.org/rep-data/index.html
Whereas, the latest OVAL definitions are here - http://oval.mitre.org/repository/data/updates/latest

There are two open source OVAL interpreters available -
Ovaldi - http://sourceforge.net/projects/ovaldi/
open-scap - http://www.open-scap.org/page/Main_Page

Since open-scap is available only for Linux and its variants and I wanted to investigate security state of a windows machine, I decided to try ovaldi.

Ovaldi installation

Download page for Ovaldi is here. Note that this links will take you to the latest version available at the post publishing time, i.e. 5.10.1. So, if there is a newer version,make use of the latest. Don't forget to change all the references from version 5.10.1 to your version in the text that follows.

Choose the EXE versions for Windows that suit your environment. In my case, it was 32-bit version, but if you have 64-bit version of Windows, download that one instead.

Unzip the file using 7-zip or winzip and and install the files to a directory - say, C:\Program Files\OVAL


Now, Ovaldi is installed!!  Also, add the file path of ovaldi.exe file to windows environment so that you do not have to type full path again and again.

If you encounter error - MSVCR100.dll is missing or

"The program can't start because MSVCR100.dll is missing from your computer. Try reinstalling the program to fix this problem."

Fix it by downloading the file from the following urls:

MSVCR100.dll = Visual C++ 2010 Runtime

32Bit: Microsoft Visual C++ 2010 SP1 Redistributable Package (x86)
http://www.microsoft.com/de-de/download/details.aspx?id=8328

64Bit: Microsoft Visual C++ 2010 SP1 Redistributable Package (x64)
http://www.microsoft.com/en-us/download/details.aspx?id=13523


Now, it is time to log-in to Windows machine as administrator and run ovaldi interpreter.


Download file definitions

Now we have interpreter and we need definitions that will be run by interpreter. Please go the the page-http://oval.mitre.org/rep-data/index.html In the page, you will see section Downloads by Version and Namespace. You need to select class to download based on the version of oval interpreter you have. The following classes are available:

  •     compliance - checks that the installation is compliant with recommended security practices.
  •     inventory - checks that produce results of what is installed.
  •     miscellaneous - misc category
  •     patch - patching status
  •     vulnerability - test that verify if there is a vulnerability present on the machine.

When you click on one of those classes you are presented with a new page that gives you a list of available definitions grouped by different criteria. For example, by clicking on vulnerability class (probably the largest one) you can select the download by platform, family or all.

For the purpose of Windows 7 testing of oval, we can downloaded file microsoft.windows.7.xml through platform/vulnerabilities, and this file was renamed to microsoft.windows.7.vulnerabilities.xml so that I do not get confused at a later stage what these tests contain!! Similarly, it is possible to downloaded equivalent files from compliance and inventory classes and you can name them as microsoft.windows.compliance.xml and microsoft.windows.inventory.xml, respectively.

Running Ovaldi

To run, please enter following command, say:

c:\program files\oval\ovaldi-5.10.1\ovaldi -m -a "c:\program files\oval\ovaldi-5.10.1\xml" -o microsoft.windows.7.vulnerability.xml -r 20150721-result.xml -x 20150721-result.html -d 20150721-system-characteristics.xml

The above command will check vulnerabilities that are present on the system. Of course, only the vulnerabilities defined in the database (microsoft.windows.7.vulnerability.xml) will checked.

An explanation for the other options is given below:
  •     Option -m. Don't check md5 sum of oval definitions file (in this case that is microsoft.windows.7.vulnerability.xml).
  •     Option -a specifies where all the auxiliary files necessary for interpreter are. For example, default style sheet file is there, also, XML definitions and tests are also there. The default value of this option assumes that you are running ovaldi in its base directory (i.e. where it is installed) so it has to be specified in order for everything to work.
  •     Option -o specifies oval definition file to use.
  •     Option -r specifies XML result file. The default value is results.xml and in the case of multiple runs, default file name will be overwritten. So, using this option prevents that from happening.
  •     Option -x specifies HTML result file. This file is generated from XML result file by applying style sheet (XSL) file. Default file is used if none is specified on the command line.
  •     Option -d specifies in which file will be saved system characteristics, i.e. installed options, existing files, etc. used during interpreter run of oval definition file.

Once the ovaldi program is finished, there will be three new files in the directory. When you open results file (20150721-result.html if you used the command given above) then you'll see four section named OVAL Results Generator Information, System Information, OVAL System Characteristics Generator Information and OVAL Definition Results.


Some links of interest related to openscap and ovaldi are given below:
  1. https://www.csiac.org/sites/default/files/vulnerability_assessment.pdf
  2. http://sgros.blogspot.in/2011/10/installing-and-testing-ovaldi-on.html
  3. http://www.vulnerabilityassessment.co.uk/ovaldi.htm

Presentation -
  1. http://nvd.nist.gov/scap/docs/conference%20presentations/workshops/OVAL%20Tutorial%201%20-%20Overview.pdf
  2. http://oval.mitre.org/community/docs/Developer_Days_2013_OVAL_Session_Minutes.pdf
  3. http://www.energy.gov/sites/prod/files/cioprod/documents/SCAP_in_Action_-_Demo_of_SCAP_Capabilities.pdf
  4. http://blog-shawndwells.rhcloud.com/wp-content/uploads/2012/07/2013-03-25-SCAP-Workshop-Coursebook.pdf
Other interesting projects related to openscap and ovaldi:
  1. MITRE course - http://benchmarkdevelopment.mitre.org/course/confirmation.html

  2. Centralized SCAP - http://blog.siphos.be/2013/09/creating-a-poor-man-central-scap-system/
     
  3. https://github.com/cyberxml/cyberxml-django





Wednesday, 17 June 2015

Python packaging - Pip trusted host issues

For quite some time,I have not updated pip. Yesterday, when I updated pip using our local PyPI server to its latest version - 7.0.3 for python2.7, my simple attempt to install python packages failed. e.g. I was trying to install fuzzy module that deals with strings similarities.

root@psj-desktop:~# pip install fuzzy -i http://osrepo.xxx.in/pypi/simpleCollecting fuzzy
  The repository located at osrepo.xxx.in is not a trusted or secure host and is being ignored. If this repository is available via HTTPS it is recommended to use HTTPS instead, otherwise you may silence this warning and allow it anyways with '--trusted-host osrepo.xxx.in'.
  Could not find a version that satisfies the requirement fuzzy (from versions: )
No matching distribution found for fuzzy

root@psj-desktop:~# pip install --pre fuzzy -i http://osrepo.xxx.in/pypi/simple
Collecting fuzzy
  The repository located at osrepo.xxx.in is not a trusted or secure host and is being ignored. If this repository is available via HTTPS it is recommended to use HTTPS instead, otherwise you may silence this warning and allow it anyways with '--trusted-host osrepo.xxx.in'.
  Could not find a version that satisfies the requirement fuzzy (from versions: )
No matching distribution found for fuzzy

Things became clear after reading pip developer documentation - https://media.readthedocs.org/pdf/pip/develop/pip.pdf
Another place you should look for is changelog in github repository - https://github.com/pypa/pip/

It is possible to create pip.conf in a number of ways:

On Unix the default configuration file is: $HOME/.config/pip/pip.conf or you can create globally under /etc/pip.conf
On Windows the configuration file is : %APPDATA%\pip\pip.ini

Sample /etc/pip.conf
---------------------------------
[global]
index-url = http://osrepo.xxx.in/pypi/simple
trusted-host = osrepo.xxx.in
disable-pip-version-check = true
allow-all-external=true
timeout = 120


Monday, 15 June 2015

Installation of Autopsy third party modules

Autopsy Forensic Browser is a graphical interface to the The Sleuth Kit and other digital investigation tools. Using both of them, you can analyze Windows and LINUX disks and file systems (NTFS, FAT, UFS1/2, Ext2/3, etc.). I was going through all the features of Autopsy on my desktop to gain first-hand experience.

A number of Autopsy modules are available here - http://wiki.sleuthkit.org/index.php?title=Autopsy_3rd_Party_Modules

For my reference, the procedure for installation of Autopsy module is given below:
  • Navigate to the latest .nbm module file - e.g. https://github.com/williballenthin/Autopsy-WindowsRegistryIngestModule/tree/master/precompiled
  • Click on the .nbm file so that the View Raw text appears.
  • Right-click on the View Raw text and select Save Link As... to save the raw .nbm file.
  • Start Autopsy and close the Welcome screen.
  • From the menu, select Tools | Plugins.
  • Open Downloaded tab and click the Add Plugins button.
  • From the Add Plugins window, navigate to the downloaded .nbm module file and open it.
  • Click Install and follow the wizard.

Tuesday, 12 May 2015

Elasticsearch error - Exception in thread ">output" org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [30s]

After adding the new elasticsearch node, initially, I was struggling with error :

WARN: org.elasticsearch.discovery: [logstash-id.xxx.in-25379-6424] waited for 30s and no initial state was set by the discovery

and I corrected the situation by adding iptables rules.

Thereafter, things appeared to be smooth but could not find new elasticsearch index using Elasticsearch head plugin even after few minutes. So, I started searching through debug logs and spotted an error :

log4j, [2015-05-12T11:00:02.003] DEBUG: org.elasticsearch.discovery.zen: [logstash-id.xxx.in-26328-4264] filtered ping responses: (filter_client[true], filter_data[false]) {none}
Exception in thread ">output" org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [30s]
    at org.elasticsearch.action.support.master.TransportMasterNodeOperationAction$3.onTimeout(org/elasticsearch/action/support/master/TransportMasterNodeOperationAction.java:180)
    at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(org/elasticsearch/cluster/service/InternalClusterService.java:492)
log4j, [2015-05-12T11:00:06.507] DEBUG: org.elasticsearch.discovery.zen: [logstash-id.xxx.in-26328-4264] filtered ping responses: (filter_client[true], filter_data[false]) {none}
^CInterrupt received. Shutting down the pipeline. {:level=>:warn, :file=>"logstash/agent.rb", :line=>"119"}

Since I was using latest version of logstash and elasticsearch, I was bit puzzled as all the google solutions(references) were pointing to the old versions of them:

[root@id admin]# /opt/logstash/bin/logstash --version
logstash 1.4.2-modified

[root@es2 ~]# rpm -qa|grep elastic -i
elasticsearch-1.4.4-1.noarch

Finally, after reading good documentation of logstash, I decided to add 'protocol' option and vola! - it worked!!

output {
        #stdout { codec => rubydebug }
        if [type] == "netflow" {
                elasticsearch {
                        cluster => "elk-cluster"
                        index => "netflow-%{+YYYY.MM.dd}"
                        host => "10.4.0.47"
                        protocol => "http"
                        workers => 2
                }
        }


Elasticsearch warning - WARN: org.elasticsearch.discovery: [...] waited for 30s and no initial state was set by the discovery

I added a new elasticsearch node and got puzzled by the error:

log4j, [2015-05-12T10:42:18.996]  WARN: org.elasticsearch.discovery: [logstash-id.xxx.in-25379-6424] waited for 30s and no initial state was set by the discovery
^C

Finally, it turned out  that iptables was blocking access to elasticsearch host. So, after adding the firewall rules, things worked flawlessly.

Allow elasticsearch in iptables(firewall) rules:

#Allow elasticsearch access
iptables -A INPUT -s 10.4.0.45 -i eth0 -p tcp --dport 9200 -j ACCEPT
iptables -A OUTPUT -d 10.4.0.45 -o eth0 -p tcp --dport 9200 -j ACCEPT