Google has open sourced Snappy, a compressing/decompressing library that processes data streams at minimum 250MB/s-500MB/s on a single core Intel Core i7 processor.
Snappy, previously known as Zippy, is a compression library used by Google in production internally by many projects including BigTable, MapReduce and RPC. Google says the library and its algorithm has been tuned for speed, and not for compression output size nor compatibility with other similar tools. Snappy has been optimized for 64-bit x86 processors, on a single core Intel Core i7 processor achieving a compression rate of at least 250MB/s and a decompression one of 500MB/s.
The high compression speed is achieved by losing on the compression ratio, the resulting output being 20-100% larger than that of other libraries, Snappy having a compression ratio of “1.5-1.7x for plain text, about 2-4x for HTML, and of course 1.0x for JPEGs, PNGs and other already-compressed data”.
Google touts Snappy as robust being “designed not to crash in the face of corrupted or malicious input”, and stable after being used to compress petabytes of data in Google’s production environment.
Snappy can be used to benchmark itself against a number of other compression libraries - zlib, LZO, LZF, FastLZ and QuickLZ –, if they are installed on the same machine. The tool comes as a C++ library that is linked into the product that is supposed to use it, but there are several other bindings for it: Haskell, Java, Perl, Python, and Ruby.
Snappy has been open sourced under the New BSD License.
