1*01826a49SYabin Cui# Parallel Zstandard (PZstandard) 2*01826a49SYabin Cui 3*01826a49SYabin CuiParallel Zstandard is a Pigz-like tool for Zstandard. 4*01826a49SYabin CuiIt provides Zstandard format compatible compression and decompression that is able to utilize multiple cores. 5*01826a49SYabin CuiIt breaks the input up into equal sized chunks and compresses each chunk independently into a Zstandard frame. 6*01826a49SYabin CuiIt then concatenates the frames together to produce the final compressed output. 7*01826a49SYabin CuiPzstandard will write a 12 byte header for each frame that is a skippable frame in the Zstandard format, which tells PZstandard the size of the next compressed frame. 8*01826a49SYabin CuiPZstandard supports parallel decompression of files compressed with PZstandard. 9*01826a49SYabin CuiWhen decompressing files compressed with Zstandard, PZstandard does IO in one thread, and decompression in another. 10*01826a49SYabin Cui 11*01826a49SYabin Cui## Usage 12*01826a49SYabin Cui 13*01826a49SYabin CuiPZstandard supports the same command line interface as Zstandard, but also provides the `-p` option to specify the number of threads. 14*01826a49SYabin CuiDictionary mode is not currently supported. 15*01826a49SYabin Cui 16*01826a49SYabin CuiBasic usage 17*01826a49SYabin Cui 18*01826a49SYabin Cui pzstd input-file -o output-file -p num-threads -# # Compression 19*01826a49SYabin Cui pzstd -d input-file -o output-file -p num-threads # Decompression 20*01826a49SYabin Cui 21*01826a49SYabin CuiPZstandard also supports piping and fifo pipes 22*01826a49SYabin Cui 23*01826a49SYabin Cui cat input-file | pzstd -p num-threads -# -c > /dev/null 24*01826a49SYabin Cui 25*01826a49SYabin CuiFor more options 26*01826a49SYabin Cui 27*01826a49SYabin Cui pzstd --help 28*01826a49SYabin Cui 29*01826a49SYabin CuiPZstandard tries to pick a smart default number of threads if not specified (displayed in `pzstd --help`). 30*01826a49SYabin CuiIf this number is not suitable, during compilation you can define `PZSTD_NUM_THREADS` to the number of threads you prefer. 31*01826a49SYabin Cui 32*01826a49SYabin Cui## Benchmarks 33*01826a49SYabin Cui 34*01826a49SYabin CuiAs a reference, PZstandard and Pigz were compared on an Intel Core i7 @ 3.1 GHz, each using 4 threads, with the [Silesia compression corpus](https://sun.aei.polsl.pl//~sdeor/index.php?page=silesia). 35*01826a49SYabin Cui 36*01826a49SYabin CuiCompression Speed vs Ratio with 4 Threads | Decompression Speed with 4 Threads 37*01826a49SYabin Cui------------------------------------------|----------------------------------- 38*01826a49SYabin Cui |  39*01826a49SYabin Cui 40*01826a49SYabin CuiThe test procedure was to run each of the following commands 2 times for each compression level, and take the minimum time. 41*01826a49SYabin Cui 42*01826a49SYabin Cui time pzstd -# -p 4 -c silesia.tar > silesia.tar.zst 43*01826a49SYabin Cui time pzstd -d -p 4 -c silesia.tar.zst > /dev/null 44*01826a49SYabin Cui 45*01826a49SYabin Cui time pigz -# -p 4 -k -c silesia.tar > silesia.tar.gz 46*01826a49SYabin Cui time pigz -d -p 4 -k -c silesia.tar.gz > /dev/null 47*01826a49SYabin Cui 48*01826a49SYabin CuiPZstandard was tested using compression levels 1-19, and Pigz was tested using compression levels 1-9. 49*01826a49SYabin CuiPigz cannot do parallel decompression, it simply does each of reading, decompression, and writing on separate threads. 50*01826a49SYabin Cui 51*01826a49SYabin Cui## Tests 52*01826a49SYabin Cui 53*01826a49SYabin CuiTests require that you have [gtest](https://github.com/google/googletest) installed. 54*01826a49SYabin CuiSet `GTEST_INC` and `GTEST_LIB` in `Makefile` to specify the location of the gtest headers and libraries. 55*01826a49SYabin CuiAlternatively, run `make googletest`, which will clone googletest and build it. 56*01826a49SYabin CuiRun `make tests && make check` to run tests. 57
