app-arch/zstd: multithreaded compression by default

[jiblime]

Integration of Clear Linux's 'multi-thread-default.patch'

By default, zstd uses one core for compression. This patch
makes zstd use all physical cores detected for compression,
increasing performance and reducing compression time.

Below's results are from using zstd's built-in benchmark,
showing a decrease of 78.13% in compression time with -T0.
The benefit is only apparent if compression is CPU-bound
and will differ based on machine and file contents.

Default (-T1)
19#linux-5.8.tar : 983869440 -> 121381009 (8.106), 4.05 MB/s ,1771.3 MB/s
zstd -T1 -b19 -i0 --priority=rt linux-5.8.tar 244.99s user 0.46s system 99% cpu 4:05.47 total

Patched default (-T0)
19#linux-5.8.tar : 983869440 -> 121384544 (8.105), 19.2 MB/s ,1756.7 MB/s
zstd -T0 -b19 -i0 --priority=rt linux-5.8.tar 297.19s user 0.63s system 554% cpu 53.692 total

Source: https://github.com/clearlinux-pkgs/zstd/blob/1.4.5-60/multi-thread-default.patch
References: #164

[jiblime]

You can test throughput compression with this script, passing 0 or 1 (or 2, etc)

#!/bin/bash

if [[ -z "${1}" ]]; then
	echo -e "Pass 1 for -T1 (default behavior), 0 for -T0 to zstd"
	exit
else
	export T="${1}"
fi

create_tmpfile() {
	fallocate -l 256MiB /tmp/zstd_testfile
	tar cf /tmp/zstd_testfile.tar zstd_testfile --remove-files
}

test_mt() {
	for x in {1..19} ; do
		time zstd -v -T"${T}" -i0 -b"${x}" --priority=rt /tmp/zstd_testfile.tar
	done
}

create_tmpfile
trap "rm /tmp/zstd_testfile.tar" EXIT

test_mt

Unrelated to the PR:

• Multithreaded compression seems to be disabled above 19

• zstd only detects physical cores, but using -T(# of logical cores) increases speed in benchmarking

[Titaniumtown]

LGTM. Good work!

[InBetweenNames]

OK this is awesome! Thank you!

[barolo]

Oh I need that! I have multiple dirs squashed with zstd, kernel is zstd and my personal archives are also zstd...

[telans]

Just need an in-kernel zstd lib update now

edit: turns out you can do that now https://github.com/facebook/zstd/tree/dev/contrib/linux-kernel#upgrading-zstd-in-the-linux-kernel