Facebook open sourced Zstandard almost six years ago with the aim of outperforming Zlib in both speed and efficiency. Zstandard 1.5 improves compression speed at intermediate compression levels, compression ratio at higher levels, and brings faster decompression speed.
Zstandard supports compression levels up to 22. Thanks to a new default match finder, Zstardard 1.5 reaches higher compression speed for levels between 5 and 12 and inputs larger than 256K. According to Facebook benchmarks, improvements range from +25% to +140% without significant loss in terms of compression ratio. Facebook claims even better results on heavily loaded machines under significant cache contention.
The biggest gains are achieved on files typically larger than 128KB. On files smaller than 16KB, by default we revert back to the legacy match finder which becomes the faster one. This default policy can be overriden manually: the new match finder can be forcibly enabled with the advanced parameter
ZSTD_c_useRowMatchFinder, or through the CLI option--[no-]row-match-finder.
At the opposite end of the compression level spectrum, Zstandard uses block splitting by default to improve compression ratio, although not by such an impressive margin as in the previously described case.
The amount of benefit varies depending on the workload. Compressing archives comprised of heavily differing files will see more improvement than compression of single files that don’t vary much entropically (like text files/enwik).
In particular, Facebook benchmarks show that Zstandard 1.5 may get up to 0.71% better compression ratio at level 22, the maximum allowed.
Zstandard 1.5 also significantly improves decompression speed, with improvements up to 21% in the best case. Actual results, though, depend largely on compiler version, payload, and compression level, says Facebook.
In general, a majority of scenarios see benefits ranging from +1 to +9%. There are also a few outliers here and there, from -4% to +13%. The average gain across all these scenarios stands at ~+4%.
Besides improving compression performance, Zstandard 1.5 builds by default with multithreaded support, standardizes a few new APIs, and deprecates a number of older ones. You can find the full detail in the official release notes.
Zstandard is based on work by Facebook engineers Yann Collet and Chip Turner. In particular, it leverages previous work by Collet based on Asymmetric numeral system (ANS).
Zstandard is integrated both within the Linux kernel, where it is used for the btrfs and squashfs filesystems, and FreeBSD, where it is used to compress core dumps. Additionally, it is used in Arch Linux and Fedora. If you want to try out Zstandard, the open-source file archiver 7zip includes support for it, along with many other codecs.