TAR: Speed up compression on multicore systems

This is a little tip to speed up creating tar archive.

By default tar using gzip to compress and if you look at htop output, gzip use only one cpu.

Tar have an argument –use-compress-program, which allow us to set another gzip-ing program.

The most known mutlithreading gzip alternatives is pigz.

So, the tar usage should looks like:

tar --use-compress-program=pigz -cf ...

Under the cut you found the compression time difference between gzip and pigz

Here’s the original tar usage:

$ tar -czf 6.5G_folder.tgz 6.5G_folder/ real 10m34.389s user 10m29.520s sys 0m13.740s

Here’s the runtime on the same system with usage of all (512) CPU

$ tar --use-compress-program=pigz -cf 6.5G_folder.tgz 6.5G_folder/ real 0m26.120s user 26m28.120s sys 0m51.490s

Leave a Reply

Your email address will not be published. Required fields are marked *