Speeding up TAR.GZ compression with PIGZ

Most of you probably noticed TAR.GZ compression isn’t very fast. Recently, during routine system backup I have realised TAR.GZ is limited not by disk read/write, but GZIP compression (98% of computation).

time sudo tar cpfz backup/ubuntu1404.tgz --one-file-system /

real 6m20.999s
user 6m1.800s
sys  0m19.043s

GZIP in its standard implementation is single CPU bound, while most of modern computers can run 4-8 threads concurrently. But there are also multi-core implementations of GZIP ie. PIGZ. I have decided to install PIGZ and plug it with TAR as follows:

sudo apt-get install lbzip2 pigz

time sudo tar cpf backup/ubuntu1404.pigz.tgz --one-file-system --use-compress-program=pigz /

real 1m43.693s
user 8m34.168s
sys  0m20.243s

As you can see, TAR.GZ using PIGZ on 4-cores i7-4770k (using 8 threads) is 2.5 times faster than GZIP! And you get standard TAR.GZ archive as output:)

The same applies to BZIP2 compression using LBZIP2.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s