replace gzip with pigz to make backups a lot faster
A user on IRC complained yesterday that we use gzip in our backup task.
He complained because gzip only uses 1 core.
To speed up the backup process on multicore machines I propose to use pigz instead of gzip.
Info about pigz: http://www.zlib.net/pigz/
It is a drop in replacement for gzip and it decreases the time needed for the backup creation a lot!
I benchmarked it a bit with my eclipse installation because i didn't have a gitlab backup lying around and got the following results:
(same commands that we use in gitlabs backup script)
Gzip:
executed command:
time tar -C ./eclipse/ -cf - . | gzip -c -1 >backup_gzip.tar
results:
real 0m17.750s
user 0m17.255s
sys 0m1.145sPigz:
executed command:
time tar -C ./eclipse/ -cf - . | pigz -c -1 >backup_pigz.tar
results:
real 0m5.635s
user 0m16.333s
sys 0m0.801sPros:
As you can see this speeds up the time considerably.
Cons:
The downside is, that we either have to bundle pigz in the omnibus install or require it as a dependency.
It is in the repos of most distributions, but not installed by default.