README.md
1## zlib-ng
2*zlib data compression library for the next generation systems*
3
4Maintained by Hans Kristian Rosbach
5 aka Dead2 (zlib-ng àt circlestorm dót org)
6
7|CI|Status|
8|:-|-|
9|GitHub Actions|[](https://github.com/zlib-ng/zlib-ng/actions) [](https://github.com/zlib-ng/zlib-ng/actions) [](https://github.com/zlib-ng/zlib-ng/actions)|
10|Buildkite|[](https://buildkite.com/circlestorm-productions/zlib-ng)|
11|CodeFactor|[](https://www.codefactor.io/repository/github/zlib-ng/zlib-ng)|
12|OSS-Fuzz|[](https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened&can=1&q=proj:zlib-ng)
13|Codecov|[](https://codecov.io/github/zlib-ng/zlib-ng/)|
14
15
16Features
17--------
18
19* Zlib compatible API with support for dual-linking
20* Modernized native API based on zlib API for ease of porting
21* Modern C11 syntax and a clean code layout
22* Deflate medium and quick algorithms based on Intel’s zlib fork
23* Support for CPU intrinsics when available
24 * Adler32 implementation using SSSE3, AVX2, AVX512, AVX512-VNNI, Neon, VMX & VSX
25 * CRC32-B implementation using PCLMULQDQ, VPCLMULQDQ, ACLE, & IBM Z
26 * Hash table implementation using CRC32-C intrinsics on x86 and ARM
27 * Slide hash implementations using SSE2, AVX2, Neon, VMX & VSX
28 * Compare256 implementations using SSE2, AVX2, Neon, & POWER9
29 * Inflate chunk copying using SSE2, AVX, Neon & VSX
30 * Support for hardware-accelerated deflate using IBM Z DFLTCC
31* Unaligned memory read/writes and large bit buffer improvements
32* Includes improvements from Cloudflare and Intel forks
33* Configure, CMake, and NMake build system support
34* Comprehensive set of CMake unit tests
35* Code sanitizers, fuzzing, and coverage
36* GitHub Actions continuous integration on Windows, macOS, and Linux
37 * Emulated CI for ARM, AARCH64, PPC, PPC64, SPARC64, S390x using qemu
38
39
40History
41-------
42
43The motivation for this fork came after seeing several 3rd party
44contributions containing new optimizations not getting implemented
45into the official zlib repository.
46
47Mark Adler has been maintaining zlib for a very long time, and he has
48done a great job and hopefully he will continue for a long time yet.
49The idea of zlib-ng is not to replace zlib, but to co-exist as a
50drop-in replacement with a lower threshold for code change.
51
52zlib has a long history and is incredibly portable, even supporting
53lots of systems that predate the Internet. This is great, but it does
54complicate further development and maintainability.
55The zlib code has numerous workarounds for old compilers that do not
56understand ANSI-C or to accommodate systems with limitations such as
57operating in a 16-bit environment.
58
59Many of these workarounds are only maintenance burdens, some of them
60are pretty huge code-wise. For example, the [v]s[n]printf workaround
61code has a whopping 8 different implementations just to cater to
62various old compilers. With this many workarounds cluttered throughout
63the code, new programmers with an idea/interest for zlib will need
64to take some time to figure out why all of these seemingly strange
65things are used, and how to work within those confines.
66
67So I decided to make a fork, merge all the Intel optimizations, merge
68the Cloudflare optimizations that did not conflict, plus a couple
69of other smaller patches. Then I started cleaning out workarounds,
70various dead code, all contrib and example code as there is little
71point in having those in this fork for various reasons.
72
73A lot of improvements have gone into zlib-ng since its start, and
74numerous people and companies have contributed both small and big
75improvements, or valuable testing.
76
77Please read LICENSE.md, it is very simple and very liberal.
78
79
80Build
81-----
82
83There are two ways to build zlib-ng:
84
85### Cmake
86
87To build zlib-ng using the cross-platform makefile generator cmake.
88
89```
90cmake .
91cmake --build . --config Release
92ctest --verbose -C Release
93```
94
95Alternatively, you can use the cmake configuration GUI tool ccmake:
96
97```
98ccmake .
99```
100
101### Configure
102
103To build zlib-ng using the bash configure script:
104
105```
106./configure
107make
108make test
109```
110
111Build Options
112-------------
113
114| CMake | configure | Description | Default |
115|:-------------------------|:-------------------------|:--------------------------------------------------------------------------------------|---------|
116| ZLIB_COMPAT | --zlib-compat | Compile with zlib compatible API | OFF |
117| ZLIB_ENABLE_TESTS | | Build test binaries | ON |
118| WITH_GZFILEOP | --without-gzfileops | Compile with support for gzFile related functions | ON |
119| WITH_OPTIM | --without-optimizations | Build with optimisations | ON |
120| WITH_NEW_STRATEGIES | --without-new-strategies | Use new strategies | ON |
121| WITH_NATIVE_INSTRUCTIONS | --native | Compiles with full instruction set supported on this host (gcc/clang -march=native) | OFF |
122| WITH_SANITIZER | | Build with sanitizer (memory, address, undefined) | OFF |
123| WITH_FUZZERS | | Build test/fuzz | OFF |
124| WITH_BENCHMARKS | | Build test/benchmarks | OFF |
125| WITH_MAINTAINER_WARNINGS | | Build with project maintainer warnings | OFF |
126| WITH_CODE_COVERAGE | | Enable code coverage reporting | OFF |
127
128
129Install
130-------
131
132WARNING: We do not recommend manually installing unless you really
133know what you are doing, because this can potentially override the system
134default zlib library, and any incompatibility or wrong configuration of
135zlib-ng can make the whole system unusable, requiring recovery or reinstall.
136If you still want a manual install, we recommend using the /opt/ path prefix.
137
138For Linux distros, an alternative way to use zlib-ng (if compiled in
139zlib-compat mode) instead of zlib, is through the use of the
140_LD_PRELOAD_ environment variable. If the program is dynamically linked
141with zlib, then zlib-ng will temporarily be used instead by the program,
142without risking system-wide instability.
143
144```
145LD_PRELOAD=/opt/zlib-ng/libz.so.1.2.11.zlib-ng /usr/bin/program
146```
147
148### Cmake
149
150To install zlib-ng system-wide using cmake:
151
152```
153cmake --build . --target install
154```
155
156### Configure
157
158To install zlib-ng system-wide using the configure script:
159
160```
161make install
162```
163
164Contributing
165------------
166
167Zlib-ng is a aiming to be open to contributions, and we would be delighted to
168receive pull requests on github.
169Just remember that any code you submit must be your own and it must be zlib licensed.
170Help with testing and reviewing of pull requests etc is also very much appreciated.
171
172If you are interested in contributing, please consider joining our
173IRC channel #zlib-ng on the Freenode IRC network.
174
175
176Acknowledgments
177----------------
178
179Thanks to Servebolt.com for sponsoring my maintainership of zlib-ng.
180
181Thanks go out to all the people and companies who have taken the time to contribute
182code reviews, testing and/or patches. Zlib-ng would not have been nearly as good without you.
183
184The deflate format used by zlib was defined by Phil Katz.
185The deflate and zlib specifications were written by L. Peter Deutsch.
186
187zlib was originally created by Jean-loup Gailly (compression)
188and Mark Adler (decompression).
189
190
191Advanced Build Options
192----------------------
193
194| CMake | configure | Description | Default |
195|:--------------------------------|:----------------------|:--------------------------------------------------------------------|------------------------|
196| ZLIB_DUAL_LINK | | Dual link tests with system zlib | OFF |
197| FORCE_SSE2 | --force-sse2 | Skip runtime check for SSE2 instructions (Always on for x86_64) | OFF (x86) |
198| FORCE_TZCNT | --force-tzcnt | Skip runtime check for TZCNT instructions | OFF |
199| WITH_AVX2 | | Build with AVX2 intrinsics | ON |
200| WITH_AVX512 | | Build with AVX512 intrinsics | ON |
201| WITH_AVX512VNNI | | Build with AVX512VNNI intrinsics | ON |
202| WITH_SSE2 | | Build with SSE2 intrinsics | ON |
203| WITH_SSE41 | | Build with SSE41 intrinsics | ON |
204| WITH_SSE42 | | Build with SSE42 intrinsics | ON |
205| WITH_PCLMULQDQ | | Build with PCLMULQDQ intrinsics | ON |
206| WITH_VPCLMULQDQ | --without-vpclmulqdq | Build with VPCLMULQDQ intrinsics | ON |
207| WITH_ACLE | --without-acle | Build with ACLE intrinsics | ON |
208| WITH_NEON | --without-neon | Build with NEON intrinsics | ON |
209| WITH_ALTIVEC | --without-altivec | Build with AltiVec (VMX) intrinsics | ON |
210| WITH_POWER8 | --without-power8 | Build with POWER8 optimisations | ON |
211| WITH_CRC32_VX | --without-crc32-vx | Build with vectorized CRC32 on IBM Z | ON |
212| WITH_DFLTCC_DEFLATE | --with-dfltcc-deflate | Build with DFLTCC intrinsics for compression on IBM Z | OFF |
213| WITH_DFLTCC_INFLATE | --with-dfltcc-inflate | Build with DFLTCC intrinsics for decompression on IBM Z | OFF |
214| WITH_UNALIGNED | --without-unaligned | Allow optimizations that use unaligned reads if safe on current arch| ON |
215| WITH_INFLATE_STRICT | | Build with strict inflate distance checking | OFF |
216| WITH_INFLATE_ALLOW_INVALID_DIST | | Build with zero fill for inflate invalid distances | OFF |
217| INSTALL_UTILS | | Copy minigzip and minideflate during install | OFF |
218
219
220Related Projects
221----------------
222
223* Fork of the popular minigzip https://github.com/zlib-ng/minizip-ng
224* Python tool to benchmark minigzip/minideflate https://github.com/zlib-ng/deflatebench
225* Python tool to benchmark pigz https://github.com/zlib-ng/pigzbench
226* 3rd party patches for zlib-ng compatibility https://github.com/zlib-ng/patches
227
){kind=link}
){kind=link}
){kind=link}
){kind=link}
){kind=link}
){kind=link}