Saturday, 24 January 2015

Installation of Clamav on CentOS

ClamAV is an open source anti-virus engine for email scanning, web scanning, and end point security.
I installed it on my CentOS machine with the following steps:

#Enable EPEL repo
#yum install clamav clamd

#chkconfig clamd on
#service clamd start

#update pattern files
#/usr/bin/freshclam

#scan
#clamscan --infected --remove --recursive /home

# test using EICAR virus
# wget http://www.eicar.org/download/eicar.com
# clamscan --infected --remove --recursive .


You can write a cron job to configure a daily scan.

clamscan.sh
------------
#!/bin/bash
DIR_SCAN="/home"
LOG_FILE="/var/log/clamav/clamscan.log"

/usr/bin/clamscan -i -r $DIR_SCAN >> $LOG_FILE


#chmod +x /etc/cron.daily/clamscan.sh

Friday, 23 January 2015

Python package installation error - pkg_resources.DistributionNotFound:xxx(package name)

If you encounter an error - "pkg_resources.DistributionNotFound:xxx(package name)" during python package installation, kindly check your setup tools version and it will most probably "setuptools 0.6rc11" or similar.

Recently, during installation of celery and flower - "a distributed task queue" packages, I encountered this error:

psj@psj-desktop:~/Downloads/nsrl-master/celery_client$ flower -h
Traceback (most recent call last):
  File "/usr/local/bin/flower", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 2655, in <module>
    working_set.require(__requires__)
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 648, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 546, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: celery


To get rid of this error, you have to upgrade existing global setuptools to the latest using:

psj@psj-desktop:~/Downloads/nsrl-master/celery_client$ sudo pip install -U distribute


If this does not work, the other option is to upgrade it to the latest version of setup tools using pypi repository.

Install latest setup tools from pypi repository
# wget https://bootstrap.pypa.io/ez_setup.py -O - | python

If this also does not work, then install latest version of setuptools using:

$ wget https://bitbucket.org/pypa/setuptools/raw/0.8/ez_setup.py
$ sudo /usr/bin/python ez_setup.py

After  this, my python package installation were successful:

#pip install celery
#pip install flower

Very nice and detailed explanations are given in Reinout's blog -





Thursday, 15 January 2015

Logging in python

I wanted to implement a python logging module which I can call from multiple python modules with my own logger settings. I was going through Python documentation on logging and I can only say, it is amazing!! You get swamped by the features offered!

Ultimately, here is how I decided to use the logging:

log.py:
-----------
#! /usr/bin/env python
import logging

def setup_logging(name):

    """ set up logging
    """

    logging.basicConfig(level=logging.DEBUG)  # (level=logging.INFO)
    #logger = logging.getLogger(__name__)
    logger = logging.getLogger(name)
  
   # set up file handler
    handler = logging.FileHandler('main.log')
    handler.setLevel(logging.DEBUG)

    # logging format
    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
    handler.setFormatter(formatter)

    # add the handlers to the logger
    logger.addHandler(handler)
    return logger

main.py:
--------------
#! /usr/bin/env python
import log
logger = log.setup_logging('root')
logger.debug('main message')

Restore MySQL database from MyISAM tables

It is assumed that you have a back up copy of MyISAM tables (.frm,.myd,.myi)

Each MyISAM table is stored on disk in three seperate files. The files have names that begin with the table name and have an extension to indicate the file type. An .frm file stores the table format. The data file has an .MYD (MYData) extension. The index file has an .MYI (MYIndex) extension.

Steps:

1) Stop running instance of MySQL Server.
2) Copy database folder containing frm,MYI and MYD files from your backup to  MySQL data directory - e.g. /var/lib/mysql
3) Change the permission of the folders to mysql.mysql so that mysql instance can read the files/directories.
4) If required, please check and repair all database tables using 'myisamchk'.

Note- While restoring the database, use the same MySQL server version.

Wednesday, 14 January 2015

Finding hash of a web page using python

I wanted to detect whether a page has changed or not. Hashing is the most common way to determine this. Using python, it turned out to be a simple four line code:

In [1]: import requests
In [2]: import hashlib
In [3]: page_contents=requests.get('http://www.google.com')
In [4]: hashlib.sha256(page_contents.text.encode('utf-8')).hexdigest()
Out[5]: 'bb5b5872d83f2f9a89912630b27dd0af145727b818f757ea86b4d6a09cadeb32'

Stack overflow source - http://stackoverflow.com/questions/17159609/create-a-checksum-for-a-fetched-webpage

Tuesday, 13 January 2015

Fixing "Unknown table engine 'InnoDB'" error

In case, you encounter an the following error after tweaking your MySQL configuration:

 Unknown table engine 'InnoDB'

The fix is simple. Do the following:

$ sudo service mysql stop
$ sudo rm /var/lib/mysql/ib_logfile*
$ sudo service mysql start

There is no harm in deleting files - ib_logfile* as the files are re-created once the database starts up.

Monday, 8 December 2014

Fix DistributionNotFound error argparse>=1.1.0 in elasticsearch-curator installation

I wanted to install elasticsearch-curator package(ver. 2.0.2) on my CentOS machine to manage my old elasticsearch(ver. 1.2.0) indices. I encountered the following weird error while installing:

$ curator --versionTraceback (most recent call last):  File "/usr/bin/curator", line 5, in <module>    from pkg_resources import load_entry_point  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2655, in <module>    working_set.require(__requires__)  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 648, in require    needed = self.resolve(parse_requirements(requirements))  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 546, in resolve    raise DistributionNotFound(req)pkg_resources.DistributionNotFound: argparse>=1.1.0

After searching on google, it became apparent that the error is peculiar to CentOS/RHEL variants only!!

Finally, I found a way..... Un-install existing installation and then re-install...

$ sudo pip uninstall argparse
$ sudo pip uninstall elasticsearch
$ sudo pip uninstall elasticsearch-curator
$ sudo pip install argparse
$ sudo pip install elasticsearch
$ sudo pip install elasticsearch-curator

Not a perfect way to solve the issue... But still, the
This thread on github might be useful - https://github.com/elasticsearch/curator/issues/56

I found another reason for this type of error: it happens when the version of setuptools and distribute libraries of python on the system are old. So, update them using pip installer:

$ sudo pip install distribute --upgrade
$ sudo pip install setuptools --upgrade