Python package installation - Unknown distribution option: 'include_package_data', 'zip_safe' etc

Packaging a python application is always tricky! The reasons are historical (setuptools vs distribute issues) and there is lot of material available on the internet to confuse you...

Anyway, I also got fumbled in packaging a python application and encountered the following error:

psj@psj-desktop:~/Downloads/nsrl$ python setup.py --help-commands
/usr/lib/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'include_package_data'
  warnings.warn(msg)
/usr/lib/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'zip_safe'
  warnings.warn(msg)
/usr/lib/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'install_requires'
  warnings.warn(msg)
Standard commands:
  build            build everything needed to install
  build_py         "build" pure Python modules (copy to build directory)
  build_ext        build C/C++ extensions (compile/link to build directory)
  build_clib       build C/C++ libraries used by Python extensions
  build_scripts    "build" scripts (copy and fixup #! line)
  clean            clean up temporary files from 'build' command
  install          install everything from build directory
  install_lib      install all Python modules (extensions and pure Python)
  install_headers  install C/C++ header files
  install_scripts  install scripts (Python or otherwise)
  install_data     install data files
  sdist            create a source distribution (tarball, zip file, etc.)
  register         register the distribution with the Python package index
  bdist            create a built (binary) distribution
  bdist_dumb       create a "dumb" built distribution
  bdist_rpm        create an RPM distribution
  bdist_wininst    create an executable installer for MS Windows
  upload           upload binary package to PyPI

After google searches, this error means that setuptools package is missing or is not at right location.

To get rid of this error, do this:

psj@psj-desktop:~ wget http://bootstrap.pypa.io/ez_setup.py -O - | sudo python
....
Processing setuptools-3.3-py2.6.egg
Copying setuptools-3.3-py2.6.egg to /usr/local/lib/python2.6/dist-packages
Adding setuptools 3.3 to easy-install.pth file
Installing easy_install script to /usr/local/bin
Installing easy_install-2.6 script to /usr/local/bin

Installed /usr/local/lib/python2.6/dist-packages/setuptools-3.3-py2.6.egg
Processing dependencies for setuptools==3.3
Finished processing dependencies for setuptools==3.3

After this, the problem disappeared.

psj@psj-desktop:~/Downloads/nsrl$ python setup.py --help-commands
Standard commands:
  build             build everything needed to install
  build_py          "build" pure Python modules (copy to build directory)
  build_ext         build C/C++ extensions (compile/link to build directory)
  build_clib        build C/C++ libraries used by Python extensions
  build_scripts     "build" scripts (copy and fixup #! line)
  clean             clean up temporary files from 'build' command


Python packaging links that I referred:

1. http://www.jeffknupp.com/blog/2013/08/16/open-sourcing-a-python-project-the-right-way/
2. https://www.digitalocean.com/community/tutorials/how-to-package-and-distribute-python-applications
3. https://packaging.python.org/en/latest/
4. http://www.scotttorborg.com/python-packaging/
5. http://pythonhosted.org/setuptools/setuptools.html#including-data-files
6. http://gehrcke.de/2014/02/distributing-a-python-command-line-application/
7. https://parijatmishra.wordpress.com/2008/10/08/python-packaging-setuptools-and-eggs/

IPython WARNING: Configuration file ipythonrc not found. Ignoring request.

ipython is an interactive shell built with python. It has amazing set of features and once you get used to it, you simply can't live without it!

Recently, some how my ipython configuration got corrupt and I was always greeted with the following message when I start an ipython shell:


psj@psj-desktop:~/Downloads/$ /usr/bin/ipython2.6 WARNING: Configuration file ipythonrc not found. Ignoring request.
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.6/IPython/ConfigLoader.py", line 66, in load
    fname = filefind(fname,incpath)
  File "/usr/lib/pymodules/python2.6/IPython/genutils.py", line 555, in filefind
    ' not found in current or supplied directories:' + `alt_dirs`
IOError: File'ipythonrc' not found in current or supplied directories:u'/home/psj/.ipython'

WARNING: Problems loading configuration file 'ipythonrc'
Starting with default -bare bones- configuration.
Python 2.6.5 (r265:79063, Feb 27 2014, 19:44:14)
Type "copyright", "credits" or "license" for more information.

IPython 0.10 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.


I tried to uninstall and install ipython module using:

$ sudo pip uninstall ipython
$ sudo pip install ipython

But, it was not of any help. I was still not able to use the amazing features of ipython. Then, I tried to locate existing 'ipythonrc' file on my system and volla! - I could find the file under /usr/share directory!

root@psj-desktop:~# locate ipythonrc
/home/psj/.ipython/ipythonrc-math
/home/psj/.ipython/ipythonrc-numeric
/home/psj/.ipython/ipythonrc-physics
/home/psj/.ipython/ipythonrc-pysh
/home/psj/.ipython/ipythonrc-tutorial
/usr/lib/pymodules/python2.6/IPython/UserConfig/ipythonrc
/usr/lib/pymodules/python2.6/IPython/UserConfig/ipythonrc-math
/usr/lib/pymodules/python2.6/IPython/UserConfig/ipythonrc-numeric
/usr/lib/pymodules/python2.6/IPython/UserConfig/ipythonrc-physics
/usr/lib/pymodules/python2.6/IPython/UserConfig/ipythonrc-pysh
/usr/lib/pymodules/python2.6/IPython/UserConfig/ipythonrc-tutorial
/usr/share/pyshared/IPython/UserConfig/ipythonrc
/usr/share/pyshared/IPython/UserConfig/ipythonrc-math
/usr/share/pyshared/IPython/UserConfig/ipythonrc-numeric
/usr/share/pyshared/IPython/UserConfig/ipythonrc-physics
/usr/share/pyshared/IPython/UserConfig/ipythonrc-pysh
/usr/share/pyshared/IPython/UserConfig/ipythonrc-tutorial

After seeing that 'ipythonrc' is missing in my  home directory, I copied it to my home directory.

root@psj-desktop:~# cp /usr/share/pyshared/IPython/UserConfig/ipythonrc /home/psj/.ipython/


And everything works nicely as usual..

psj@psj-desktop:~$ sudo ln -s /usr/bin/ipython2.6 /usr/bin/ipython
psj@psj-desktop:~$ ipython
Python 2.6.5 (r265:79063, Feb 27 2014, 19:44:14)
Type "copyright", "credits" or "license" for more information.

IPython 0.10 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

In [1]: import os

In [2]: os.
Display all 234 possibilities? (y or n)

Installation of Clamav on CentOS

ClamAV is an open source anti-virus engine for email scanning, web scanning, and end point security.
I installed it on my CentOS machine with the following steps:

#Enable EPEL repo
#yum install clamav clamd

#chkconfig clamd on
#service clamd start

#update pattern files
#/usr/bin/freshclam

#scan
#clamscan --infected --remove --recursive /home

# test using EICAR virus
# wget http://www.eicar.org/download/eicar.com
# clamscan --infected --remove --recursive .


You can write a cron job to configure a daily scan.

clamscan.sh
------------
#!/bin/bash
DIR_SCAN="/home"
LOG_FILE="/var/log/clamav/clamscan.log"

/usr/bin/clamscan -i -r $DIR_SCAN >> $LOG_FILE


#chmod +x /etc/cron.daily/clamscan.sh

Python package installation error - pkg_resources.DistributionNotFound:xxx(package name)

If you encounter an error - "pkg_resources.DistributionNotFound:xxx(package name)" during python package installation, kindly check your setup tools version and it will most probably "setuptools 0.6rc11" or similar.

Recently, during installation of celery and flower - "a distributed task queue" packages, I encountered this error:

psj@psj-desktop:~/Downloads/nsrl-master/celery_client$ flower -h
Traceback (most recent call last):
  File "/usr/local/bin/flower", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 2655, in <module>
    working_set.require(__requires__)
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 648, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 546, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: celery


To get rid of this error, you have to upgrade existing global setuptools to the latest using:

psj@psj-desktop:~/Downloads/nsrl-master/celery_client$ sudo pip install -U distribute


If this does not work, the other option is to upgrade it to the latest version of setup tools using pypi repository.

Install latest setup tools from pypi repository
# wget https://bootstrap.pypa.io/ez_setup.py -O - | python

If this also does not work, then install latest version of setuptools using:

$ wget https://bitbucket.org/pypa/setuptools/raw/0.8/ez_setup.py
$ sudo /usr/bin/python ez_setup.py

After  this, my python package installation were successful:

#pip install celery
#pip install flower

Very nice and detailed explanations are given in Reinout's blog -

Logging in python

I wanted to implement a python logging module which I can call from multiple python modules with my own logger settings. I was going through Python documentation on logging and I can only say, it is amazing!! You get swamped by the features offered!

Ultimately, here is how I decided to use the logging:

log.py:
-----------
#! /usr/bin/env python
import logging

def setup_logging(name):

    """ set up logging
    """

    logging.basicConfig(level=logging.DEBUG)  # (level=logging.INFO)
    #logger = logging.getLogger(__name__)
    logger = logging.getLogger(name)
  
   # set up file handler
    handler = logging.FileHandler('main.log')
    handler.setLevel(logging.DEBUG)

    # logging format
    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
    handler.setFormatter(formatter)

    # add the handlers to the logger
    logger.addHandler(handler)
    return logger

main.py:
--------------
#! /usr/bin/env python
import log
logger = log.setup_logging('root')
logger.debug('main message')

Restore MySQL database from MyISAM tables

It is assumed that you have a back up copy of MyISAM tables (.frm,.myd,.myi)

Each MyISAM table is stored on disk in three seperate files. The files have names that begin with the table name and have an extension to indicate the file type. An .frm file stores the table format. The data file has an .MYD (MYData) extension. The index file has an .MYI (MYIndex) extension.

Steps:

1) Stop running instance of MySQL Server.
2) Copy database folder containing frm,MYI and MYD files from your backup to  MySQL data directory - e.g. /var/lib/mysql
3) Change the permission of the folders to mysql.mysql so that mysql instance can read the files/directories.
4) If required, please check and repair all database tables using 'myisamchk'.

Note- While restoring the database, use the same MySQL server version.

Finding hash of a web page using python

I wanted to detect whether a page has changed or not. Hashing is the most common way to determine this. Using python, it turned out to be a simple four line code:

In [1]: import requests
In [2]: import hashlib
In [3]: page_contents=requests.get('http://www.google.com')
In [4]: hashlib.sha256(page_contents.text.encode('utf-8')).hexdigest()
Out[5]: 'bb5b5872d83f2f9a89912630b27dd0af145727b818f757ea86b4d6a09cadeb32'

Stack overflow source - http://stackoverflow.com/questions/17159609/create-a-checksum-for-a-fetched-webpage

Fixing "Unknown table engine 'InnoDB'" error

In case, you encounter an the following error after tweaking your MySQL configuration:

 Unknown table engine 'InnoDB'

The fix is simple. Do the following:

$ sudo service mysql stop
$ sudo rm /var/lib/mysql/ib_logfile*
$ sudo service mysql start

There is no harm in deleting files - ib_logfile* as the files are re-created once the database starts up.