BIGTOP-3921: Add ZSTD Codec Support for hadoop

[rzuo]

Description of PR

Add ZSTD codec compiled to hadoop native library by default.

How was this patch tested?

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'BIGTOP-3638. Your PR title ...')?
• Make sure that newly added files do not have any licensing issues. When in doubt refer to https://www.apache.org/licenses/

[iwasakims]

We need fix for toolchain too to support this on all supported distros/platforms.

[iwasakims]

Is the Zstandard licensed under the one we can bundle it by default?
https://www.apache.org/legal/resolved.html

[rzuo]

We need fix for toolchain too to support this on all supported distros/platforms.

In bigtop_toolchain/manifests/packages.pp, libzstd-devel has been included, so anything else need to modify?

Thanks
Robin

[rzuo]

Is the Zstandard licensed under the one we can bundle it by default? https://www.apache.org/legal/resolved.html

By my understanding, this fix only make the hadoop native library compiled with pre-installed ZSTD binary, no ZSTD (GPL-2.0 license) code included in this repository.
If no zstd library present, this script will compile as before.

Thanks
Robin

[guyuqi]

Is the Zstandard licensed under the one we can bundle it by default? https://www.apache.org/legal/resolved.html

zsdt's license is BSD (https://github.com/facebook/zstd/blob/dev/LICENSE).
As far as I know, Apache 2.0 license is compatible with BSD. (Pls correct me if I had misunderstanding)

[rzuo]

Is the Zstandard licensed under the one we can bundle it by default? https://www.apache.org/legal/resolved.html

zsdt's license is BSD (https://github.com/facebook/zstd/blob/dev/LICENSE). As far as I know, Apache 2.0 license is compatible with BSD. (Pls correct me if I had misunderstanding)

aha, you are correct, i miss understood that zstd is GPL licensed.

[iwasakims]

In bigtop_toolchain/manifests/packages.pp, libzstd-devel has been included, so anything else need to modify?

Oops. I forgot about BIGTOP-3535.

[iwasakims]

By my understanding, this fix only make the hadoop native library compiled with pre-installed ZSTD binary, no ZSTD (GPL-2.0 license) code included in this repository.

We are publishing pre built packages for users' convenience. We can not link it against pre-built library if it is distributed under GPL. Zstandard seems to be dual-licensed under BSD and GPLv2. I'm not confident about which is applied to pre-built libzstd.so redistributed by OS distros.

[iwasakims]

Can we expect all supported distro/platform have libzstd.so? We can add package dependency on libzstd to make it certain that the required library is installed to runtime environment if so.

[rzuo]

emmm, I only have CentOS 7.4 at hand.
Can someone help to verify other distros/platforms?

[guyuqi]

emmm, I only have CentOS 7.4 at hand. Can someone help to verify other distros/platforms?

Assume your development environment is x86 64/amd64-based.
Please try to test it in various Distros of Bigtop docker images for x86 64/amd64 first.
(https://hub.docker.com/r/bigtop/slaves/tags)

[rzuo]

Ok, I'll paste the result later, thanks.

[rzuo]

With bigtop docker images for x86_64, got following result:

1. centos-7

2. debian-11

3. ubuntu-22.04

4. fedora-36

5. rockylinux-8

[guyuqi]

By my understanding, this fix only make the hadoop native library compiled with pre-installed ZSTD binary, no ZSTD (GPL-2.0 license) code included in this repository.

We are publishing pre built packages for users' convenience. We can not link it against pre-built library if it is distributed under GPL. Zstandard seems to be dual-licensed under BSD and GPLv2. I'm not confident about which is applied to pre-built libzstd.so redistributed by OS distros.

I could completely understand your concerns for LICENSE. The terms of the license are too complicated to comprehend well.
Fortunately, it appears that Apache Spark does support ZSTD and Spark guys had updated LICENSE and NOTICE in SPARK-24654.
They provided the extra license statement in source tree: LICENSE-binary-zstd.
Shall we also create the similar LICENSE file for zstd binary in Bigtop?

What do you think of it? @iwasakims

[iwasakims]

Spark bundles pre-built binary of zstd-jni (containing zstd as part of it). I'm concerning about the license of libzstd binary distributed by OS distros here.