Update zstd to version 1.5.0

[SirSniper]

Pull latest C code from https://github.com/facebook/zstd/

This is the code as of https://github.com/facebook/zstd/tree/v1.5.0 (facebook/zstd@a488ba1)

Similar to #73 (comment) to pull changes with edits to flatten the directory structure

Benchmark on a Windows (i7-7700k 4.2GHz w/ 32GB) for mr from http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia:

benchstat old.txt new.txt
name                   old time/op    new time/op     delta
CtxCompression-8         92.6ms ± 2%     85.4ms ± 2%   -7.68%  (p=0.000 n=9+9)
CtxDecompression-8       9.44ms ± 1%     9.46ms ± 1%     ~     (p=0.315 n=9+10)
StreamCompression-8       105ms ± 3%       99ms ± 2%   -5.55%  (p=0.000 n=8+10)
StreamDecompression-8    13.7ms ± 9%     12.6ms ± 4%   -7.75%  (p=0.022 n=10+9)
Compression-8             109ms ±12%       86ms ± 2%  -21.00%  (p=0.000 n=9+9)
Decompression-8          10.0ms ± 4%      9.5ms ± 2%   -4.24%  (p=0.001 n=10+9)

name                   old speed      new speed       delta
CtxCompression-8        108MB/s ± 2%    117MB/s ± 2%   +8.33%  (p=0.000 n=9+9)
CtxDecompression-8     1.06GB/s ± 1%   1.05GB/s ± 1%     ~     (p=0.286 n=9+10)
StreamCompression-8    94.7MB/s ± 3%  100.3MB/s ± 2%   +5.88%  (p=0.000 n=8+10)
StreamDecompression-8   733MB/s ±10%    791MB/s ± 4%   +7.96%  (p=0.022 n=10+9)
Compression-8          91.7MB/s ±11%  115.6MB/s ± 2%  +26.05%  (p=0.000 n=9+9)
Decompression-8        1.00GB/s ± 4%   1.05GB/s ± 2%   +4.35%  (p=0.001 n=10+9)

Looks like overall performance improvements across the board.

[Viq111]

Thanks for the PR! Do you mind double checking what commit you pulled the files from ?
Looking at facebook/zstd@a488ba1 , there are some changes in this PR that are not in 1.5.0 (example: https://github.com/facebook/zstd/blob/v1.5.0/lib/common/bitstream.h#L334-L335) but are in there

I'm hypothesizing that you might have pulled files from the dev branch instead of v1.5.0 (but it changes a lot so hard to know what commit you pulled from.

Do you mind pulling the files from facebook/zstd@a488ba1 instead ?

[SirSniper]

That appears to be exactly what happened. I just pulled the latest code from facebook/zstd@a488ba1 and committed to the branch. Reran my performance results

name                   old time/op    new time/op     delta
CtxCompression-8         87.1ms ± 2%     75.9ms ± 2%  -12.80%  (p=0.000 n=10+9)
CtxDecompression-8       9.60ms ± 1%     9.74ms ± 1%   +1.46%  (p=0.001 n=10+8)
StreamCompression-8       102ms ± 2%       87ms ± 2%  -15.02%  (p=0.000 n=9+9)
StreamDecompression-8    13.1ms ± 5%     12.9ms ± 3%     ~     (p=0.400 n=9+10)
Compression-8            88.1ms ± 3%     76.5ms ± 1%  -13.20%  (p=0.000 n=10+8)
Decompression-8          9.77ms ± 3%     9.93ms ± 4%     ~     (p=0.393 n=10+10)

name                   old speed      new speed       delta
CtxCompression-8        115MB/s ± 2%    131MB/s ± 2%  +14.67%  (p=0.000 n=10+9)
CtxDecompression-8     1.04GB/s ± 1%   1.02GB/s ± 1%   -1.44%  (p=0.001 n=10+8)
StreamCompression-8    97.6MB/s ± 2%  114.8MB/s ± 2%  +17.67%  (p=0.000 n=9+9)
StreamDecompression-8   753MB/s ± 9%    771MB/s ± 3%     ~     (p=0.247 n=10+10)
Compression-8           113MB/s ± 3%    130MB/s ± 1%  +15.19%  (p=0.000 n=10+8)
Decompression-8        1.02GB/s ± 3%   1.01GB/s ± 4%     ~     (p=0.393 n=10+10)

name                   old alloc/op   new alloc/op    delta
CtxCompression-8          56.0B ± 0%      56.0B ± 0%     ~     (all equal)
CtxDecompression-8        64.0B ± 0%      64.0B ± 0%     ~     (all equal)
StreamCompression-8      1.34MB ± 0%     1.07MB ± 4%  -19.63%  (p=0.000 n=8+10)
StreamDecompression-8    7.21kB ±24%     5.76kB ±10%     ~     (p=0.064 n=10+8)
Compression-8             48.0B ± 0%      49.8B ± 8%     ~     (p=0.294 n=8+10)
Decompression-8           48.0B ± 0%      48.0B ± 0%     ~     (all equal)

name                   old allocs/op  new allocs/op   delta
CtxCompression-8           3.00 ± 0%       3.00 ± 0%     ~     (all equal)
CtxDecompression-8         3.00 ± 0%       3.00 ± 0%     ~     (all equal)
StreamCompression-8        3.00 ± 0%       3.00 ± 0%     ~     (all equal)
StreamDecompression-8       161 ± 0%        161 ± 0%     ~     (all equal)
Compression-8              2.00 ± 0%       2.00 ± 0%     ~     (all equal)
Decompression-8            2.00 ± 0%       2.00 ± 0%     ~     (all equal)

[Viq111]

That looks great, thanks for upgrading to 1.5.0!

The results with the payload mr from silesia corpus) are actually great on Mac:

goos: darwin
goarch: amd64
pkg: github.com/DataDog/zstd
cpu: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz

name                    old time/op    new time/op     delta
CtxCompression-16         92.9ms ± 1%     70.6ms ± 1%  -23.99%  (p=0.008 n=5+5)
CtxDecompression-16       10.9ms ± 1%     10.3ms ± 1%   -5.42%  (p=0.008 n=5+5)
StreamCompression-16       102ms ± 1%       80ms ± 1%  -21.13%  (p=0.008 n=5+5)
StreamDecompression-16    13.0ms ± 1%     12.5ms ± 2%   -3.77%  (p=0.008 n=5+5)
Compression-16            92.6ms ± 0%     71.2ms ± 1%  -23.19%  (p=0.008 n=5+5)
Decompression-16          10.8ms ± 0%     10.3ms ± 2%   -4.51%  (p=0.008 n=5+5)

name                    old speed      new speed       delta
CtxCompression-16        107MB/s ± 1%    141MB/s ± 1%  +31.56%  (p=0.008 n=5+5)
CtxDecompression-16      915MB/s ± 1%    967MB/s ± 1%   +5.72%  (p=0.008 n=5+5)
StreamCompression-16    97.8MB/s ± 1%  124.0MB/s ± 1%  +26.79%  (p=0.008 n=5+5)
StreamDecompression-16   767MB/s ± 1%    797MB/s ± 2%   +3.92%  (p=0.008 n=5+5)
Compression-16           108MB/s ± 0%    140MB/s ± 1%  +30.19%  (p=0.008 n=5+5)
Decompression-16         925MB/s ± 0%    969MB/s ± 2%   +4.74%  (p=0.008 n=5+5)

(-20% on Compression, -~3-5% on Decompression)