Using docker for application development

I found Docker super useful, but going through a manual is quite time consuming. Here, very stripped manual to create your first image and push it online 🙂

[bash]
# install docker
wget -qO- https://get.docker.com/ | sh

# add your user to docker group
sudo usermod -aG docker $USER

# check if it’s working
docker run docker/whalesay cowsay "hello world!"

# create an account on https://hub.docker.com
# and login
docker login -u $USER –email=EMAIL

# run image
docker run -it ubuntu

# make some changes ie. create user, install needed software etc

# finally open new terminal & commit changes (SESSIONID=HOSTNAME)
docker commit SESSIONID $USER/image:version

# mount local directory `pwd`/test as /test in read/write mode
docker run -it -v `pwd`/test:/test:rw $USER/image:version some command with arguments

# push image
docker push $USER/image:version
[/bash]

From now, you can get your image from any other machine connected to Internet by executing:
[bash]
docker run -it $USER/image:version
# ie. redundans image
docker run -it -w /root/src/redundans lpryszcz/redundans:v0.11b ./redundans.py -v -i test/{600,5000}_{1,2}.fq.gz -f test/contigs.fa -o test/run1

# you can create alias latest, then version can be skipped on running
docker tag lpryszcz/redundans:v0.11b lpryszcz/redundans:latest
docker push lpryszcz/redundans:latest

docker run -it lpryszcz/redundans
[/bash]

You can add info about your repository at https://hub.docker.com/r/$USER/image/

Advertisements

Installing new version of Python without root

Some time ago I was recommending to use Python virtual environment to install local version of Python packages. However this will not solve the issue of outdated version Python in the server your are working in. Here, pythonbrew may be help for you.

[bash]
# install pythonbrew to ~/.pythonbrew
curl -kL http://xrl.us/pythonbrewinstall | bash

# add to ~/.bashrc to automatically activate pythonbrew
[[ -s "$HOME/.pythonbrew/etc/bashrc" ]] && source "$HOME/.pythonbrew/etc/bashrc"

# open new terminal tab (Ctrl+Shift+T) or window (Ctrl+Shift+N)

# install python 2.7.10
pythonbrew install 2.7.10

# and enable the new version
pythonbrew switch 2.7.10

# from now on, you can enjoy the version of your choice and install dependencies
which python
#/home/…/.pythonbrew/pythons/Python-2.7.10/bin/python
python –version
#Python 2.7.10
which pip
#/home/…/.pythonbrew/pythons/Python-2.7.10/bin/pip
[/bash]

Using pybedtools under Python 2.6

I encountered an import error from collections import OrderedDict, while using pybedtools under Python 2.6. It took me some time to find a workaround… I posted it below.
[bash]
# install dependencies and pybedtools
pip install cython ordereddict
pip install pybedtools

# edit file … lib/python2.6/site-packages/pybedtools/contrib/venn_maker.py
# comment import from collections and add try-except
#from collections import OrderedDict
try:
from collections import OrderedDict
except ImportError:
from ordereddict import OrderedDict
[/bash]

Virtual environment (venv) in Python

Working on the machines with no root access is sometimes annoying, especially if you depend on multiple Python packages and your distro is somehow outdated… You may find Python virtual environment (venv) very useful.
First, you need to create venv directory structure (this is done only once):
[bash]
mkdir -p ~/src/venv
cd ~/src/venv
virtualenv py27
[/bash]

Then you can open new BASH terminal and activate your venv by:
[bash]source ~/src/venv/py27/bin/activate[/bash]

After that, you can install / upgrade any packages using pip / easy_install (even including PIP ) ie.
[bash]
pip install –upgrade pip
pip install –upgrade scipy
[/bash]

Insipired by python-guide.

Apache2 reading from sshfs share

Today, I have encountered problems trying to read data from sshfs share in apache2. I was getting 403 Forbidden error. It turned out you need to enable other_user in sshfs, so other users than the one mounting the share can access the data, as apache2 is using www-data user.
[bash]
# uncomment last line of /etc/fuse.conf
# Allow non-root users to specify the allow_other or allow_root mount options.
user_allow_other

# enable other_user and read access by non-root
sudo chmod a+r /etc/fuse.conf

# remount
sudo umount DESTINATION
sshfs -o allow_other SHARE DESTINATION
[/bash]

Inspired by serverfault and unix.stackexchange.

Easy install to specific folder on machines without root

I’m using more & more remote machines in which I don’t have admin rights, so usual sudo easy_install isn’t working. But there is simple work around by installing Python packages to non-default directory.
[bash]
# prepare new Python packages directory
mkdir -p ~/src/python/site-packages

# edit ~/.bashrc so Python knows where to look for packages
export PYTHONPATH=$PYTHONPATH:~/src:~/src/python:~/src/python/site-packages

# install packages
easy_install –install-dir /home/USER/src/python/site-packages PACKAGE_NAME

# ie for pysam installation you should do
easy_install –install-dir /home/lpryszcz/src/python/site-packages cython pyrex pysam
[/bash]