This package provides various compression algorithms.
- zstandard compression and decompression in pure Go.
- S2 is a high performance replacement for Snappy.
- Optimized deflate packages which can be used as a dropin replacement for gzip, zip and zlib.
- snappy is a drop-in replacement for
github.com/golang/snappyoffering better compression and concurrent streams. - huff0 and FSE implementations for raw entropy encoding.
- gzhttp Provides client and server wrappers for handling gzipped requests efficiently.
- pgzip is a separate package that provides a very fast parallel gzip implementation.
Use go get github.com/klauspost/compress@latest to add it to your project.
This package will support the current Go version and 2 versions back.
- Use the
nounsafetag to disable all use of the "unsafe" package. - Use the
noasmtag to disable all assembly across packages.
Use the links above for more information on each.
-
Feb 19th, 2025 - 1.18.0
- Add unsafe little endian loaders #1036
- fix: check
r.err != nilbut return a nil value errorerrby @alingse in #1028 - flate: Simplify L4-6 loading #1043
- flate: Simplify matchlen (remove asm) #1045
- s2: Improve small block compression speed w/o asm #1048
- flate: Fix matchlen L5+L6 #1049
- flate: Cleanup & reduce casts #1050
-
Oct 11th, 2024 - 1.17.11
-
Sep 23rd, 2024 - 1.17.10
-
Jun 12th, 2024 - 1.17.9
-
Apr 9th, 2024 - 1.17.8
-
Feb 21st, 2024 - 1.17.7
-
Feb 5th, 2024 - 1.17.6
-
Jan 26th, 2024 - v1.17.5
- flate: Fix reset with dictionary on custom window encodes #912
- zstd: Add Frame header encoding and stripping #908
- zstd: Limit better/best default window to 8MB #913
- zstd: Speed improvements by @greatroar in #896 #910
- s2: Fix callbacks for skippable blocks and disallow 0xfe (Padding) by @Jille in #916 #917 #919 #918
-
Dec 1st, 2023 - v1.17.4
-
Nov 15th, 2023 - v1.17.3
-
Oct 22nd, 2023 - v1.17.2
- zstd: Fix rare CORRUPTION output in "best" mode. See #876
-
Oct 14th, 2023 - v1.17.1
-
Sept 19th, 2023 - v1.17.0
See changes to v1.16.x
-
July 1st, 2023 - v1.16.7
-
June 13, 2023 - v1.16.6
-
Apr 16, 2023 - v1.16.5
-
Apr 5, 2023 - v1.16.4
- zstd: Improve zstd best efficiency by @greatroar and @klauspost in #784
- zstd: Respect WithAllLitEntropyCompression #792
- zstd: Fix amd64 not always detecting corrupt data #785
- zstd: Various minor improvements by @greatroar in #788 #794 #795
- s2: Fix huge block overflow #779
- s2: Allow CustomEncoder fallback #780
- gzhttp: Support ResponseWriter Unwrap() in gzhttp handler by @jgimenez in #799
-
Mar 13, 2023 - v1.16.1
-
Feb 26, 2023 - v1.16.0
See changes to v1.15.x
-
Jan 21st, 2023 (v1.15.15)
-
Jan 3rd, 2023 (v1.15.14)
-
Dec 11, 2022 (v1.15.13)
-
Oct 26, 2022 (v1.15.12)
-
Sept 26, 2022 (v1.15.11)
-
Sept 16, 2022 (v1.15.10)
- zstd: Add WithDecodeAllCapLimit #649
- Add Go 1.19 - deprecate Go 1.16 #651
- flate: Improve level 5+6 compression #656
- zstd: Improve "better" compression #657
- s2: Improve "best" compression #658
- s2: Improve "better" compression. #635
- s2: Slightly faster non-assembly decompression #646
- Use arrays for constant size copies #659
-
July 21, 2022 (v1.15.9)
-
July 13, 2022 (v1.15.8)
- gzip: fix stack exhaustion bug in Reader.Read #641
- s2: Add Index header trim/restore #638
- zstd: Optimize seqdeq amd64 asm by @greatroar in #636
- zstd: Improve decoder memcopy #637
- huff0: Pass a single bitReader pointer to asm by @greatroar in #634
- zstd: Branchless getBits for amd64 w/o BMI2 by @greatroar in #640
- gzhttp: Remove header before writing #639
-
June 29, 2022 (v1.15.7)
-
June 3, 2022 (v1.15.6)
- s2: Improve coding for long, close matches #613
- s2c: Add Snappy/S2 stream recompression #611
- zstd: Always use configured block size #605
- zstd: Fix incorrect hash table placement for dict encoding in default #606
- zstd: Apply default config to ZipDecompressor without options #608
- gzhttp: Exclude more common archive formats #612
- s2: Add ReaderIgnoreCRC #609
- s2: Remove sanity load on index creation #607
- snappy: Use dedicated function for scoring #614
- s2c+s2d: Use official snappy framed extension #610
-
May 25, 2022 (v1.15.5)
- s2: Add concurrent stream decompression #602
- s2: Fix final emit oob read crash on amd64 #601
- huff0: asm implementation of Decompress1X by @WojciechMula #596
- zstd: Use 1 less goroutine for stream decoding #588
- zstd: Copy literal in 16 byte blocks when possible #592
- zstd: Speed up when WithDecoderLowmem(false) #599
- zstd: faster next state update in BMI2 version of decode by @WojciechMula in #593
- huff0: Do not check max size when reading table. #586
- flate: Inplace hashing for level 7-9 #590
-
May 11, 2022 (v1.15.4)
-
May 5, 2022 (v1.15.3)
-
Apr 26, 2022 (v1.15.2)
-
Mar 11, 2022 (v1.15.1)
-
Mar 3, 2022 (v1.15.0)
Both compression and decompression now supports "synchronous" stream operations. This means that whenever "concurrency" is set to 1, they will operate without spawning goroutines.
Stream decompression is now faster on asynchronous, since the goroutine allocation much more effectively splits the workload. On typical streams this will typically use 2 cores fully for decompression. When a stream has finished decoding no goroutines will be left over, so decoders can now safely be pooled and still be garbage collected.
While the release has been extensively tested, it is recommended to testing when upgrading.
See changes to v1.14.x
-
Feb 22, 2022 (v1.14.4)
-
Feb 17, 2022 (v1.14.3)
-
Jan 25, 2022 (v1.14.2)
-
Jan 11, 2022 (v1.14.1)
See changes to v1.13.x
-
Aug 30, 2021 (v1.13.5)
-
Aug 12, 2021 (v1.13.4)
- Add snappy replacement package.
- zstd: Fix incorrect encoding in "best" mode #415
-
Aug 3, 2021 (v1.13.3)
-
Jun 14, 2021 (v1.13.1)
- s2: Add full Snappy output support #396
- zstd: Add configurable Decoder window size #394
- gzhttp: Add header to skip compression #389
- s2: Improve speed with bigger output margin #395
-
Jun 3, 2021 (v1.13.0)
See changes to v1.12.x
-
May 25, 2021 (v1.12.3)
-
Apr 27, 2021 (v1.12.2)
-
Apr 14, 2021 (v1.12.1)
- snappy package removed. Upstream added as dependency.
- s2: Better compression in "best" mode #353
- s2sx: Add stdin input and detect pre-compressed from signature #352
- s2c/s2d: Add http as possible input #348
- s2c/s2d/s2sx: Always truncate when writing files #352
- zstd: Reduce memory usage further when using WithLowerEncoderMem #346
- s2: Fix potential problem with amd64 assembly and profilers #349
See changes to v1.11.x
-
Mar 26, 2021 (v1.11.13)
-
Mar 5, 2021 (v1.11.12)
- s2: Add
s2sxbinary that creates self extracting archives. - s2: Speed up decompression on non-assembly platforms #328
- s2: Add
-
Mar 1, 2021 (v1.11.9)
-
Feb 25, 2021 (v1.11.8)
-
Jan 14, 2021 (v1.11.7)
-
Jan 7, 2021 (v1.11.6)
-
Dec 20, 2020 (v1.11.4)
-
Nov 15, 2020 (v1.11.3)
-
Oct 11, 2020 (v1.11.2)
- s2: Fix out of bounds read in "better" block compression #291
-
Oct 1, 2020 (v1.11.1)
- zstd: Set allLitEntropy true in default configuration #286
-
Sept 8, 2020 (v1.11.0)
- zstd: Add experimental compression dictionaries #281
- zstd: Fix mixed Write and ReadFrom calls #282
- inflate/gz: Limit variable shifts, ~5% faster decompression #274
See changes to v1.10.x
-
July 8, 2020 (v1.10.11)
-
June 23, 2020 (v1.10.10)
- zstd: Skip entropy compression in fastest mode when no matches. #270
-
June 16, 2020 (v1.10.9):
-
June 5, 2020 (v1.10.8):
- 1.15x faster zstd block decompression. #265
-
June 1, 2020 (v1.10.7):
- Added zstd decompression dictionary support
- Increase zstd decompression speed up to 1.19x. #259
- Remove internal reset call in zstd compression and reduce allocations. #263
-
May 21, 2020: (v1.10.6)
-
April 12, 2020: (v1.10.5)
- s2-commands: Flush output when receiving SIGINT. #239
-
Apr 8, 2020: (v1.10.4)
-
Mar 11, 2020: (v1.10.3)
-
Feb 27, 2020: (v1.10.2)
-
Feb 18, 2020: (v1.10.1)
-
Feb 4, 2020: (v1.10.0)
- Add optional dictionary to stateless deflate. Breaking change, send
nilfor previous behaviour. #216 - Fix buffer overflow on repeated small block deflate. #218
- Allow copying content from an existing ZIP file without decompressing+compressing. #214
- Added S2 AMD64 assembler and various optimizations. Stream speed >10GB/s. #186
- Add optional dictionary to stateless deflate. Breaking change, send
See changes prior to v1.10.0
- Jan 20,2020 (v1.9.8) Optimize gzip/deflate with better size estimates and faster table generation. #207 by luyu6056, #206.
- Jan 11, 2020: S2 Encode/Decode will use provided buffer if capacity is big enough. #204
- Jan 5, 2020: (v1.9.7) Fix another zstd regression in v1.9.5 - v1.9.6 removed.
- Jan 4, 2020: (v1.9.6) Regression in v1.9.5 fixed causing corrupt zstd encodes in rare cases.
- Jan 4, 2020: Faster IO in s2c + s2d commandline tools compression/decompression. #192
- Dec 29, 2019: Removed v1.9.5 since fuzz tests showed a compatibility problem with the reference zstandard decoder.
- Dec 29, 2019: (v1.9.5) zstd: 10-20% faster block compression. #199
- Dec 29, 2019: zip package updated with latest Go features
- Dec 29, 2019: zstd: Single segment flag condintions tweaked. #197
- Dec 18, 2019: s2: Faster compression when ReadFrom is used. #198
- Dec 10, 2019: s2: Fix repeat length output when just above at 16MB limit.
- Dec 10, 2019: zstd: Add function to get decoder as io.ReadCloser. #191
- Dec 3, 2019: (v1.9.4) S2: limit max repeat length. #188
- Dec 3, 2019: Add WithNoEntropyCompression to zstd #187
- Dec 3, 2019: Reduce memory use for tests. Check for leaked goroutines.
- Nov 28, 2019 (v1.9.3) Less allocations in stateless deflate.
- Nov 28, 2019: 5-20% Faster huff0 decode. Impacts zstd as well. #184
- Nov 12, 2019 (v1.9.2) Added Stateless Compression for gzip/deflate.
- Nov 12, 2019: Fixed zstd decompression of large single blocks. #180
- Nov 11, 2019: Set default s2c block size to 4MB.
- Nov 11, 2019: Reduce inflate memory use by 1KB.
- Nov 10, 2019: Less allocations in deflate bit writer.
- Nov 10, 2019: Fix inconsistent error returned by zstd decoder.
- Oct 28, 2019 (v1.9.1) ztsd: Fix crash when compressing blocks. #174
- Oct 24, 2019 (v1.9.0) zstd: Fix rare data corruption #173
- Oct 24, 2019 zstd: Fix huff0 out of buffer write #171 and always return errors #172
- Oct 10, 2019: Big deflate rewrite, 30-40% faster with better compression #105
See changes prior to v1.9.0
- Oct 10, 2019: (v1.8.6) zstd: Allow partial reads to get flushed data. #169
- Oct 3, 2019: Fix inconsistent results on broken zstd streams.
- Sep 25, 2019: Added
-rm(remove source files) and-q(no output except errors) tos2cands2dcommands - Sep 16, 2019: (v1.8.4) Add
s2cands2dcommandline tools. - Sep 10, 2019: (v1.8.3) Fix s2 decoder Skip.
- Sep 7, 2019: zstd: Added WithWindowSize, contributed by ianwilkes.
- Sep 5, 2019: (v1.8.2) Add WithZeroFrames which adds full zero payload block encoding option.
- Sep 5, 2019: Lazy initialization of zstandard predefined en/decoder tables.
- Aug 26, 2019: (v1.8.1) S2: 1-2% compression increase in "better" compression mode.
- Aug 26, 2019: zstd: Check maximum size of Huffman 1X compressed literals while decoding.
- Aug 24, 2019: (v1.8.0) Added S2 compression, a high performance replacement for Snappy.
- Aug 21, 2019: (v1.7.6) Fixed minor issues found by fuzzer. One could lead to zstd not decompressing.
- Aug 18, 2019: Add fuzzit continuous fuzzing.
- Aug 14, 2019: zstd: Skip incompressible data 2x faster. #147
- Aug 4, 2019 (v1.7.5): Better literal compression. #146
- Aug 4, 2019: Faster zstd compression. #143 #144
- Aug 4, 2019: Faster zstd decompression. #145 #143 #142
- July 15, 2019 (v1.7.4): Fix double EOF block in rare cases on zstd encoder.
- July 15, 2019 (v1.7.3): Minor speedup/compression increase in default zstd encoder.
- July 14, 2019: zstd decoder: Fix decompression error on multiple uses with mixed content.
- July 7, 2019 (v1.7.2): Snappy update, zstd decoder potential race fix.
- June 17, 2019: zstd decompression bugfix.
- June 17, 2019: fix 32 bit builds.
- June 17, 2019: Easier use in modules (less dependencies).
- June 9, 2019: New stronger "default" zstd compression mode. Matches zstd default compression ratio.
- June 5, 2019: 20-40% throughput in zstandard compression and better compression.
- June 5, 2019: deflate/gzip compression: Reduce memory usage of lower compression levels.
- June 2, 2019: Added zstandard compression!
- May 25, 2019: deflate/gzip: 10% faster bit writer, mostly visible in lower levels.
- Apr 22, 2019: zstd decompression added.
- Aug 1, 2018: Added huff0 README.
- Jul 8, 2018: Added Performance Update 2018 below.
- Jun 23, 2018: Merged Go 1.11 inflate optimizations. Go 1.9 is now required. Backwards compatible version tagged with v1.3.0.
- Apr 2, 2018: Added huff0 en/decoder. Experimental for now, API may change.
- Mar 4, 2018: Added FSE Entropy en/decoder. Experimental for now, API may change.
- Nov 3, 2017: Add compression Estimate function.
- May 28, 2017: Reduce allocations when resetting decoder.
- Apr 02, 2017: Change back to official crc32, since changes were merged in Go 1.7.
- Jan 14, 2017: Reduce stack pressure due to array copies. See Issue #18625.
- Oct 25, 2016: Level 2-4 have been rewritten and now offers significantly better performance than before.
- Oct 20, 2016: Port zlib changes from Go 1.7 to fix zlib writer issue. Please update.
- Oct 16, 2016: Go 1.7 changes merged. Apples to apples this package is a few percent faster, but has a significantly better balance between speed and compression per level.
- Mar 24, 2016: Always attempt Huffman encoding on level 4-7. This improves base 64 encoded data compression.
- Mar 24, 2016: Small speedup for level 1-3.
- Feb 19, 2016: Faster bit writer, level -2 is 15% faster, level 1 is 4% faster.
- Feb 19, 2016: Handle small payloads faster in level 1-3.
- Feb 19, 2016: Added faster level 2 + 3 compression modes.
- Feb 19, 2016: Rebalanced compression levels, so there is a more even progression in terms of compression. New default level is 5.
- Feb 14, 2016: Snappy: Merge upstream changes.
- Feb 14, 2016: Snappy: Fix aggressive skipping.
- Feb 14, 2016: Snappy: Update benchmark.
- Feb 13, 2016: Deflate: Fixed assembler problem that could lead to sub-optimal compression.
- Feb 12, 2016: Snappy: Added AMD64 SSE 4.2 optimizations to matching, which makes easy to compress material run faster. Typical speedup is around 25%.
- Feb 9, 2016: Added Snappy package fork. This version is 5-7% faster, much more on hard to compress content.
- Jan 30, 2016: Optimize level 1 to 3 by not considering static dictionary or storing uncompressed. ~4-5% speedup.
- Jan 16, 2016: Optimization on deflate level 1,2,3 compression.
- Jan 8 2016: Merge CL 18317: fix reading, writing of zip64 archives.
- Dec 8 2015: Make level 1 and -2 deterministic even if write size differs.
- Dec 8 2015: Split encoding functions, so hashing and matching can potentially be inlined. 1-3% faster on AMD64. 5% faster on other platforms.
- Dec 8 2015: Fixed rare one byte out-of bounds read. Please update!
- Nov 23 2015: Optimization on token writer. ~2-4% faster. Contributed by @dsnet.
- Nov 20 2015: Small optimization to bit writer on 64 bit systems.
- Nov 17 2015: Fixed out-of-bound errors if the underlying Writer returned an error. See #15.
- Nov 12 2015: Added io.WriterTo support to gzip/inflate.
- Nov 11 2015: Merged CL 16669: archive/zip: enable overriding (de)compressors per file
- Oct 15 2015: Added skipping on uncompressible data. Random data speed up >5x.
The packages are drop-in replacements for standard libraries. Simply replace the import path to use them:
Typical speed is about 2x of the standard library packages.
| old import | new import | Documentation |
|---|---|---|
compress/gzip |
github.com/klauspost/compress/gzip |
gzip |
compress/zlib |
github.com/klauspost/compress/zlib |
zlib |
archive/zip |
github.com/klauspost/compress/zip |
zip |
compress/flate |
github.com/klauspost/compress/flate |
flate |
You may also be interested in pgzip, which is a drop in replacement for gzip, which support multithreaded compression on big files and the optimized crc32 package used by these packages.
The packages contains the same as the standard library, so you can use the godoc for that: gzip, zip, zlib, flate.
Currently there is only minor speedup on decompression (mostly CRC32 calculation).
Memory usage is typically 1MB for a Writer. stdlib is in the same range. If you expect to have a lot of concurrently allocated Writers consider using the stateless compress described below.
For compression performance, see: this spreadsheet.
To disable all assembly add -tags=noasm. This works across all packages.
This package offers stateless compression as a special option for gzip/deflate. It will do compression but without maintaining any state between Write calls.
This means there will be no memory kept between Write calls, but compression and speed will be suboptimal.
This is only relevant in cases where you expect to run many thousands of compressors concurrently, but with very little activity. This is not intended for regular web servers serving individual requests.
Because of this, the size of actual Write calls will affect output size.
In gzip, specify level -3 / gzip.StatelessCompression to enable.
For direct deflate use, NewStatelessWriter and StatelessDeflate are available. See documentation
A bufio.Writer can of course be used to control write sizes. For example, to use a 4KB buffer:
// replace 'ioutil.Discard' with your output.
gzw, err := gzip.NewWriterLevel(ioutil.Discard, gzip.StatelessCompression)
if err != nil {
return err
}
defer gzw.Close()
w := bufio.NewWriterSize(gzw, 4096)
defer w.Flush()
// Write to 'w' This will only use up to 4KB in memory when the writer is idle.
Compression is almost always worse than the fastest compression level and each write will allocate (a little) memory.
Here are other packages of good quality and pure Go (no cgo wrappers or autoconverted code):
- github.com/pierrec/lz4 - strong multithreaded LZ4 compression.
- github.com/cosnicolaou/pbzip2 - multithreaded bzip2 decompression.
- github.com/dsnet/compress - brotli decompression, bzip2 writer.
- github.com/ronanh/intcomp - Integer compression.
- github.com/spenczar/fpc - Float compression.
- github.com/minio/zipindex - External ZIP directory index.
- github.com/ybirader/pzip - Fast concurrent zip archiver and extractor.
This code is licensed under the same conditions as the original Go code. See LICENSE file.
