Working with millions of intermediate files can be very challenging, especially if you need to store them in distributed / network file system (NFS). This will make listing / navigating the directories to take ages… and removing of these files very time-consuming.
During building metaPhOrs DB, I needed to store some ~7.5 million of intermediate files that were subsequently processed in HPC. Saving these amount of files in the NFS would seriously affect not only myself, but also overall system performance.
One could store files in an archive, but then if you want to retrieve the data you would need to parse rather huge archives (tens-to-hundreds of GB) in order to retrieve rather small portions of data.
I have realised that TAR archives are natively supported in Python and can be indexed (see `tar_indexer`), which provide easy integration into existing code and random-access. If you work with text data, you can even zlib.compress the data stored inside you archives!
Below, I’m providing relevant parts of my code:
# index content of multiple tar archives
tar2index.py -v -i db_*/*.tar -d archives.db3
# search for some_file in mutliple archives
tar2index.py -v -f some_file -d archives.db3
import sqlite3, time
import tarfile, zlib, cStringIO
# lookup function
def tar_lookup(dbpath, file_name):
"""Return file name inside tar, tar file name, offset and file size."""
cur = sqlite3.connect(dbpath).cursor()
cur.execute("""SELECT o.file_name, f.file_name, offset, file_size
FROM offset_data as o JOIN file_data as f ON o.file_id=f.file_id
WHERE o.file_name like ?""", (file_name,))
# saving to archive
# open tarfile
tar = tarfile.open(tarpath, "w")
# save files to tar
for fname, txt in files_generator:
# compress file content (optionally)
gztxt = zlib.compress(txt)
# get tarinfo
ti = tarfile.TarInfo(fname)
ti.size = len(gztxt)
ti.mtime = time.time()
# add to tar
# reading from indexed archive(s)
# NOTE: before you need to run tar2index.py on your archives
tarfnames = tar_lookup(index_path, file_name)
for i, (name, tarfn, offset, file_size) in enumerate(tarfnames, 1):
tarf = open(tarfn)
# move pointer to right archive place
# read tar fragment & uncompress
txt = zlib.decompress(tarf.read(file_size))