Wednesday, 17 June 2015

Python packaging - Pip trusted host issues

For quite some time,I have not updated pip. Yesterday, when I updated pip using our local PyPI server to its latest version - 7.0.3 for python2.7, my simple attempt to install python packages failed. e.g. I was trying to install fuzzy module that deals with strings similarities.

root@psj-desktop:~# pip install fuzzy -i http://osrepo.xxx.in/pypi/simpleCollecting fuzzy
  The repository located at osrepo.xxx.in is not a trusted or secure host and is being ignored. If this repository is available via HTTPS it is recommended to use HTTPS instead, otherwise you may silence this warning and allow it anyways with '--trusted-host osrepo.xxx.in'.
  Could not find a version that satisfies the requirement fuzzy (from versions: )
No matching distribution found for fuzzy

root@psj-desktop:~# pip install --pre fuzzy -i http://osrepo.xxx.in/pypi/simple
Collecting fuzzy
  The repository located at osrepo.xxx.in is not a trusted or secure host and is being ignored. If this repository is available via HTTPS it is recommended to use HTTPS instead, otherwise you may silence this warning and allow it anyways with '--trusted-host osrepo.xxx.in'.
  Could not find a version that satisfies the requirement fuzzy (from versions: )
No matching distribution found for fuzzy

Things became clear after reading pip developer documentation - https://media.readthedocs.org/pdf/pip/develop/pip.pdf
Another place you should look for is changelog in github repository - https://github.com/pypa/pip/

It is possible to create pip.conf in a number of ways:

On Unix the default configuration file is: $HOME/.config/pip/pip.conf or you can create globally under /etc/pip.conf
On Windows the configuration file is : %APPDATA%\pip\pip.ini

Sample /etc/pip.conf
---------------------------------
[global]
index-url = http://osrepo.xxx.in/pypi/simple
trusted-host = osrepo.xxx.in
disable-pip-version-check = true
allow-all-external=true
timeout = 120


Monday, 15 June 2015

Installation of Autopsy third party modules

Autopsy Forensic Browser is a graphical interface to the The Sleuth Kit and other digital investigation tools. Using both of them, you can analyze Windows and LINUX disks and file systems (NTFS, FAT, UFS1/2, Ext2/3, etc.). I was going through all the features of Autopsy on my desktop to gain first-hand experience.

A number of Autopsy modules are available here - http://wiki.sleuthkit.org/index.php?title=Autopsy_3rd_Party_Modules

For my reference, the procedure for installation of Autopsy module is given below:
  • Navigate to the latest .nbm module file - e.g. https://github.com/williballenthin/Autopsy-WindowsRegistryIngestModule/tree/master/precompiled
  • Click on the .nbm file so that the View Raw text appears.
  • Right-click on the View Raw text and select Save Link As... to save the raw .nbm file.
  • Start Autopsy and close the Welcome screen.
  • From the menu, select Tools | Plugins.
  • Open Downloaded tab and click the Add Plugins button.
  • From the Add Plugins window, navigate to the downloaded .nbm module file and open it.
  • Click Install and follow the wizard.

Tuesday, 12 May 2015

Elasticsearch error - Exception in thread ">output" org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [30s]

After adding the new elasticsearch node, initially, I was struggling with error :

WARN: org.elasticsearch.discovery: [logstash-id.xxx.in-25379-6424] waited for 30s and no initial state was set by the discovery

and I corrected the situation by adding iptables rules.

Thereafter, things appeared to be smooth but could not find new elasticsearch index using Elasticsearch head plugin even after few minutes. So, I started searching through debug logs and spotted an error :

log4j, [2015-05-12T11:00:02.003] DEBUG: org.elasticsearch.discovery.zen: [logstash-id.xxx.in-26328-4264] filtered ping responses: (filter_client[true], filter_data[false]) {none}
Exception in thread ">output" org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [30s]
    at org.elasticsearch.action.support.master.TransportMasterNodeOperationAction$3.onTimeout(org/elasticsearch/action/support/master/TransportMasterNodeOperationAction.java:180)
    at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(org/elasticsearch/cluster/service/InternalClusterService.java:492)
log4j, [2015-05-12T11:00:06.507] DEBUG: org.elasticsearch.discovery.zen: [logstash-id.xxx.in-26328-4264] filtered ping responses: (filter_client[true], filter_data[false]) {none}
^CInterrupt received. Shutting down the pipeline. {:level=>:warn, :file=>"logstash/agent.rb", :line=>"119"}

Since I was using latest version of logstash and elasticsearch, I was bit puzzled as all the google solutions(references) were pointing to the old versions of them:

[root@id admin]# /opt/logstash/bin/logstash --version
logstash 1.4.2-modified

[root@es2 ~]# rpm -qa|grep elastic -i
elasticsearch-1.4.4-1.noarch

Finally, after reading good documentation of logstash, I decided to add 'protocol' option and vola! - it worked!!

output {
        #stdout { codec => rubydebug }
        if [type] == "netflow" {
                elasticsearch {
                        cluster => "elk-cluster"
                        index => "netflow-%{+YYYY.MM.dd}"
                        host => "10.4.0.47"
                        protocol => "http"
                        workers => 2
                }
        }


Elasticsearch warning - WARN: org.elasticsearch.discovery: [...] waited for 30s and no initial state was set by the discovery

I added a new elasticsearch node and got puzzled by the error:

log4j, [2015-05-12T10:42:18.996]  WARN: org.elasticsearch.discovery: [logstash-id.xxx.in-25379-6424] waited for 30s and no initial state was set by the discovery
^C

Finally, it turned out  that iptables was blocking access to elasticsearch host. So, after adding the firewall rules, things worked flawlessly.

Allow elasticsearch in iptables(firewall) rules:

#Allow elasticsearch access
iptables -A INPUT -s 10.4.0.45 -i eth0 -p tcp --dport 9200 -j ACCEPT
iptables -A OUTPUT -d 10.4.0.45 -o eth0 -p tcp --dport 9200 -j ACCEPT

Elasticsearch curator - delete commands

If you wish to delete some old elasticsearch indices, curator is the de-facto standard. The program provides so many useful options and I appreciate the versatility. Although, curator has a wiki and elasticsearch page, things were not that clear to me in one go. So, here are my notes on deleting elasticsearch indices:

[admin@mgr ~]$ curator --version
curator, version 3.0.0

[admin@mgr ~]$ /usr/bin/curator --host 10.4.0.46 delete --help
Usage: curator delete [OPTIONS] COMMAND [ARGS]...

  Delete indices or snapshots

Options:
  --disk-space FLOAT  Delete indices beyond DISK_SPACE gigabytes.
  --reverse BOOLEAN   Only valid with --disk-space. Affects sort order of the
                      indices.  True means reverse-alphabetical (if dates are
                      involved, older is deleted first).  [default: True]
  --help              Show this message and exit.

Commands:
  indices    Index selection.
  snapshots  Snapshot selection.


[admin@mgr ~]$ /usr/bin/curator --host 10.4.0.46 delete indices --help
Usage: curator delete indices [OPTIONS]

  Get a list of indices to act on from the provided arguments, then perform
  the command [alias, allocation, bloom, close, delete, etc.] on the
  resulting list.

Options:
  --newer-than INTEGER            Include only indices newer than n time_units
  --older-than INTEGER            Include only indices older than n time_units
  --prefix TEXT                   Include only indices beginning with prefix.
  --suffix TEXT                   Include only indices ending with suffix.
  --time-unit [hours|days|weeks|months]
                                  Unit of time to reckon by
  --timestring TEXT               Python strftime string to match your index
                                  definition, e.g. 2014.07.15 would be
                                  %Y.%m.%d
  --regex TEXT                    Provide your own regex, e.g
                                  '^prefix-.*-suffix$'
  --exclude TEXT                  Exclude matching indices. Can be invoked
                                  multiple times.
  --index TEXT                    Include the provided index in the list. Can
                                  be invoked multiple times.
  --all-indices                   Do not filter indices.  Act on all indices.
  --help                          Show this message and exit.

If you wish to use dry-run feature in curator:

[admin@mgr ~]$ /usr/bin/curator --host 10.4.0.46 --dry-run --debug delete indices --time-unit days --timestring "\%Y.\%m.\%d" --older-than 7 --prefix logstash- --all-indices

You should always specify logfile and loglevel (CRITICAL = 50, ERROR=40, WARNING=30, INFO=20, DEBUG=10, NOTSET=0) to capture curator program output for ease of debugging and specify sufficient timeout period. The timeout period becomes important when you are taking snapshots of indices. So, always  give sufficient timeout period:

[admin@mgr ~]$ /usr/bin/curator --host 10.4.0.47 --timeout 300 --logfile curator_log.txt --loglevel 10 --debug delete indices --time-unit days --timestring "\%Y.\%m.\%d" --older-than 7 --prefix logstash- --all-indices


[admin@mgr ~]$ /usr/bin/curator --host 10.4.0.46 --timeout 300 --logfile curator_log.txt --loglevel 10 --debug delete indices --time-unit days --timestring "\%Y.\%m.\%d" --older-than 7 --prefix logstash-apache --all-indices

If you wish to delete the indices on the basis of hard disk space,use the following command:

[admin@mgr ~]$ /usr/bin/curator --host 10.4.0.46 --timeout 300 --logfile ttt.txt --loglevel 10 --debug delete --disk-space 40 indices --all-indices




Thursday, 30 April 2015

Logstash - require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar

After installation of logstash 1.4.2 rpm on CentOS, I got stuck across this error:

[root@psj admin]# tail -f /var/log/logstash/logstash.err
  require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
  require at org/jruby/RubyKernel.java:1085
   (root) at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/ffi.rb:1
   (root) at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:1
  require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
  require at org/jruby/RubyKernel.java:1085
     LibC at /opt/logstash/lib/logstash/util/prctl.rb:4
   (root) at /opt/logstash/lib/logstash/util/prctl.rb:3
     main at /opt/logstash/lib/logstash/runner.rb:79
   (root) at /opt/logstash/lib/logstash/runner.rb:215
^C

[root@psj admin]# service logstash status
logstash is not running


After some time, I realized that it may be a path issue  or not getting proper java environment variables issue.
So, to get rid of this, please check your environment for JAVA paths:

[root@psj ELK]# cat /etc/environment
JRE_HOME=/usr/java/jre1.8.0_40/
JAVA_HOME=/usr/java/jdk1.8.0_31
JDK_HOME=/usr/java/jdk1.8.0_31/
[root@psj ELK]# source /etc/environment

Also, add logstash and java to the path:

[root@psj ELK]# export PATH=$PATH:/usr/java/default/bin
[root@psj ELK]# export PATH=/opt/logstash/bin:$PATH

[root@psj ELK]# service logstash status
logstash is running
If this is not your case, perhaps, you can look up bug details reported on github.
https://github.com/elastic/logstash/issues/1289

Logstash - LoadError: Could not load FFI Provider: (NotImplementedError) FFI not available: null

If you get error "LoadError: Could not load FFI Provider" while running logstash(ver 1.4.2) daemon on CentOS like:


[root@psj ELK]# cat /var/log/logstash/logstash.err
LoadError: Could not load FFI Provider: (NotImplementedError) FFI not available: null
 See http://jira.codehaus.org/browse/JRUBY-4583
  require at org/jruby/RubyKernel.java:1085
  require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
   (root) at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/ffi/ffi.rb:69
  require at org/jruby/RubyKernel.java:1085
   (root) at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:1
  require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
  require at org/jruby/RubyKernel.java:1085
   (root) at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/ffi.rb:1
   (root) at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:1
  require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
  require at org/jruby/RubyKernel.java:1085
     LibC at /opt/logstash/lib/logstash/util/prctl.rb:4
   (root) at /opt/logstash/lib/logstash/util/prctl.rb:3
     main at /opt/logstash/lib/logstash/runner.rb:79
   (root) at /opt/logstash/lib/logstash/runner.rb:215


Please do the following:

Modify "LS_JAVA_OPTS" option in /etc/sysconfig/logstash file  as given below:

#vim /etc/sysconfig/logstash
...
#LS_JAVA_OPTS="-Djava.io.tmpdir=$HOME"
#LS_JAVA_OPTS="-Djava.net.preferIPv4Stack=true -Djava.io.tmpdir=$HOME"
LS_JAVA_OPTS="-Djava.net.preferIPv4Stack=true -Djava.io.tmpdir=/var/lib/logstash"
...

"Djava.io.tmpdir" can be set to any directory of your preference instead of "/var/lib/logstash".

[root@psj ELK]# service logstash status
logstash is running