Using docker for application development

I found Docker super useful, but going through a manual is quite time consuming. Here, very stripped manual to create your first image and push it online 🙂

# install docker
wget -qO- | sh

# add your user to docker group
sudo usermod -aG docker $USER

# check if it’s working
docker run docker/whalesay cowsay "hello world!"

# create an account on
# and login
docker login -u $USER –email=EMAIL

# run image
docker run -it ubuntu

# make some changes ie. create user, install needed software etc

# finally open new terminal & commit changes (SESSIONID=HOSTNAME)
docker commit SESSIONID $USER/image:version

# mount local directory `pwd`/test as /test in read/write mode
docker run -it -v `pwd`/test:/test:rw $USER/image:version some command with arguments

# push image
docker push $USER/image:version

From now, you can get your image from any other machine connected to Internet by executing:
docker run -it $USER/image:version
# ie. redundans image
docker run -it -w /root/src/redundans lpryszcz/redundans:v0.11b ./ -v -i test/{600,5000}_{1,2}.fq.gz -f test/contigs.fa -o test/run1

# you can create alias latest, then version can be skipped on running
docker tag lpryszcz/redundans:v0.11b lpryszcz/redundans:latest
docker push lpryszcz/redundans:latest

docker run -it lpryszcz/redundans

You can add info about your repository at$USER/image/

Working efficiently with millions of files

Working with millions of intermediate files can be very challenging, especially if you need to store them in distributed / network file system (NFS). This will make listing / navigating the directories to take ages… and removing of these files very time-consuming.
During building metaPhOrs DB, I needed to store some ~7.5 million of intermediate files that were subsequently processed in HPC. Saving these amount of files in the NFS would seriously affect not only myself, but also overall system performance.
One could store files in an archive, but then if you want to retrieve the data you would need to parse rather huge archives (tens-to-hundreds of GB) in order to retrieve rather small portions of data.
I have realised that TAR archives are natively supported in Python and can be indexed (see `tar_indexer`), which provide easy integration into existing code and random-access. If you work with text data, you can even zlib.compress the data stored inside you archives!
Below, I’m providing relevant parts of my code:
# index content of multiple tar archives -v -i db_*/*.tar -d archives.db3

# search for some_file in mutliple archives -v -f some_file -d archives.db3

import sqlite3, time
import tarfile, zlib, cStringIO

# lookup function
def tar_lookup(dbpath, file_name):
"""Return file name inside tar, tar file name, offset and file size."""
cur = sqlite3.connect(dbpath).cursor()
cur.execute("""SELECT o.file_name, f.file_name, offset, file_size
FROM offset_data as o JOIN file_data as f ON o.file_id=f.file_id
WHERE o.file_name like ?""", (file_name,))
return cur.fetchall()

# saving to archive
# open tarfile
tar =, "w")
# save files to tar
for fname, txt in files_generator:
# compress file content (optionally)
gztxt = zlib.compress(txt)
# get tarinfo
ti = tarfile.TarInfo(fname)
ti.size = len(gztxt)
ti.mtime = time.time()
# add to tar
tar.addfile(ti, cStringIO.StringIO(gztxt))

# reading from indexed archive(s)
# NOTE: before you need to run on your archives
tarfnames = tar_lookup(index_path, file_name)
for i, (name, tarfn, offset, file_size) in enumerate(tarfnames, 1):
tarf = open(tarfn)
# move pointer to right archive place
# read tar fragment & uncompress
txt = zlib.decompress(

Tracing exceptions in multiprocessing in Python

I had problems with debugging my programme using multiprocessing.Pool.
Traceback (most recent call last):
File "src/", line 294, in <module>
File "src/", line 289, in main
o.noupload, o.verbose)
File "src/", line 242, in homologies2mysql
for i, data in enumerate(p.imap_unordered(worker, pairs), 1):
File "/usr/lib64/python2.6/multiprocessing/", line 520, in next
raise value
ValueError: need more than 1 value to unpack

I could run it without multiprocessing, but then I’d have to wait some days for the program to reach the point where it crashes.
Luckily, Python is equipped with traceback, that allows handy tracing of exceptions.
Then, you can add a decorator to problematic function, that will report nice error message:
import traceback, functools, multiprocessing

def trace_unhandled_exceptions(func):
def wrapped_func(*args, **kwargs):
return func(*args, **kwargs)
print ‘Exception in ‘+func.__name__
return wrapped_func

def go():
raise Exception()

p = multiprocessing.Pool(1)


The error message will look like:
Exception in go
Traceback (most recent call last):
File "<stdin>", line 5, in wrapped_func
File "<stdin>", line 4, in go

Solution found on StackOverflow.

Connecting to MySQL without passwd prompt

If you are (like me) annoyed by providing password at every mysql login, you can skip it. Also it makes easier programmatic access to any MySQL db, as not passwd prompting is necessary 🙂
Create `~/.my.cnf` file:



And login without `-p` parameter:
mysql -h host -u username dbname

If you want to use `~/.my.cnf` file in MySQLdb, just connect using this:
import MySQLdb
cnx = MySQLdb.connect(host=host, port=port, read_default_file="~/.my.cnf")

Batch convert of .xlsx (Microsoft Office) to .tsv (tab-delimited) files

I had to retrieve data from multiple .xlsx files with multiple sheets. This can be done manually, but it will be rather time-consuming tasks, plus Office quotes text fields, which is not very convenient for downstream analysis…
I have found handy script,, that does the job, but it reports only one sheet at the time. Thus, I have rewritten a little to save all sheets from given .xlsx file into separate folder. In addition, multiple .xlsx files can be process at once. My version can be found on github.
[bash] *.xlsx

Installing Gene Cluster on Ubuntu

Gene Cluster is a program for clustering. I wanted to use it to analyse gene expression data. However, I had problems during installation under Ubuntu 14.04. This is how I solved it:
# install dependencies: Motif libraries
sudo apt-get install libxext-dev libmotif-dev

Get GeneCluster3.0 source code and unpack it.
# configure to install in local dir
./configure –prefix=`pwd` –program-prefix=gene_ && make && make install

# add install dir to ~/.bashrc

Identification of potential transcription factor binding sites (TFBS) across species

My colleague asked me for help with identification of targets for some transcription factors (TFs). The complication is that target motifs for these TFs are known in human, (A/G)GGTGT(C/G/T)(A/G), but exact binding motif is not known in the species of interest. Nevertheless, we decided to scan the genome for matches of this motifs. To facilitate that, I’ve written small program,, finding sequence motifs in the genome. The program employs regex to find matches in forward and reverse complement and reports bed-formatted output.
[bash] -vcs -i DANRE.fa -r "[AG]GGTGT[CGT][AG]" > tf.bed 2> tf.log
[/bash] is quite fast, scanning 1.5G genome in ~1 minute on modern desktop. The program reports some basic stats ie. number of matches in +/- strand for each chromosome to stderr.
Most likely, you will find hundred thousands of putative TFBS. Therefore, it’s good to filter some of them ie. focusing on these in proximity of some genes of interest. This can be accomplished using combination of awk, bedtools and two other scripts: and
# crosslink with genes within 100 kb upstream of coding genes
awk ‘$3=="gene"’ genome.gtf > gene.gtf
cat tf.bed | 100000 | bedtools intersect -s -loj -a – -b gene.gtf | > tf.genes100k.bed

And this is how example output will look like:

1       68669   68677   GGGTGTGG        0       +       ENSDARG00000034862; ENSDARG00000088581; ENSDARG00000100782; ENSDARG00000076900; ENSDARG00000075827; ENSDARG00000096578  f7; f10; F7 (4 of 4); PROZ (2 of 2); f7i; cul4a
1       71354   71362   aggtgtgg        0       +       ENSDARG00000034862; ENSDARG00000088581; ENSDARG00000100181; ENSDARG00000100782; ENSDARG00000076900; ENSDARG00000075827; ENSDARG00000096578      f7; f10; LAMP1 (2 of 2); F7 (4 of 4); PROZ (2 of 2); f7i; cul4a
1       76322   76330   AGGTGTGG        0       +       ENSDARG00000034862; ENSDARG00000088581; ENSDARG00000100181; ENSDARG00000100782; ENSDARG00000076900; ENSDARG00000075827; ENSDARG00000096578      f7; f10; LAMP1 (2 of 2); F7 (4 of 4); PROZ (2 of 2); f7i; cul4a

All above mentioned programs can be found in github.
If you want to learn more about regular expression, have a look at Python re module.