No zlib in WASI

[DaAwesomeP]

Bug report
When compiling the WASI version using the docker image, it does not appear to include zlib in the build, or maybe the run-python-wasi.sh is not configured to allow zlib to work: ModuleNotFoundError: No module named 'zlib'

If this is already a known issue, then I can open a pull to update the docs.

Your environment

• CPython versions tested on: main 30610d2
• Operating system and architecture: quay.io/tiran/cpythonbuild:emsdk3 x86_64

[DaAwesomeP]

Reading more docs, this may be really because --enable-wasm-dynamic-linking is not supported on WASI?

[tiran]

WASI-SDK does not come with zlib out of the box. My container image does not contain any extra libraries for WASI yet. I'll look into external dependencies once we have buildbots working.

[robbertvanginkel]

I ran into this as well when trying to use the precompiled stdlib from make wasm_stdlib in a wasi build. This would fail as the script that produces the stdlib compresses with zlib:

cpython/Tools/wasm/wasm_assets.py

Lines 221 to 224 in 1de4395

	# bpo-17004: zipimport supports only zlib compression.
	# Emscripten ZIP_STORED + -sLZ4=1 linker flags results in larger file.
	args.compression = zipfile.ZIP_DEFLATED
	args.compresslevel = 9

I managed to build my own wasi python with zlib with the following dockerfile:

# syntax=docker/dockerfile:1.4

# must be linux/amd64 as wasi-sdk only has amd64 binaries
FROM --platform=linux/amd64 debian:bullseye
RUN apt-get update \
 && apt-get install -y xz-utils git libxml2 build-essential pkg-config zlib1g-dev curl python3 \
 && rm -rf /var/lib/apt/lists/*
ADD https://github.com/python/cpython/archive/refs/tags/v3.11.1.tar.gz /src/python-v3.11.1.tar.gz
ADD https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-16/wasi-sdk-16.0-linux.tar.gz /src/
ADD https://www.zlib.net/zlib-1.2.13.tar.xz /src/
ADD https://github.com/bytecodealliance/wasmtime/releases/download/v4.0.0/wasmtime-v4.0.0-x86_64-linux.tar.xz /tmp/wasmtime.tar.xz
WORKDIR /build
RUN mkdir /opt/wasi-sdk && tar -xf /src/wasi-sdk-16.0-linux.tar.gz --strip-components=1 -C /opt/wasi-sdk/ \
    	&& tar -xf /tmp/wasmtime.tar.xz --strip-components=1 -C /usr/local/bin \
	&& tar -xf /src/python-v3.11.1.tar.gz \
	&& tar -xf /src/zlib-1.2.13.tar.xz
RUN <<EOF
set -e
export PATH="/opt/wasi-sdk/bin:$PATH"
cd zlib-1.2.13/
CC=/opt/wasi-sdk/bin/clang RANLIB=/opt/wasi-sdk/bin/ranlib ./configure --prefix=
make install \
	prefix=/opt/wasi-sdk/share/wasi-sysroot \
	libdir=/opt/wasi-sdk/share/wasi-sysroot/lib/wasm32-wasi \
	pkgconfigdir=/opt/wasi-sdk/share/wasi-sysroot/lib/pkgconfig
cd ../cpython-3.11.1
./Tools/wasm/wasm_build.py wasi
/opt/wasi-sdk/bin/llvm-strip /build/cpython-3.11.1/builddir/wasi/python.wasm
cd builddir/wasi
make wasm_stdlib
EOF

With that, I can run an interpreter with the compressed stdlib:

$ docker run -it <ref to build docker file above>
root@4b26fe2dbb1b:/build# cd cpython-3.11.1/builddir/wasi/
root@4b26fe2dbb1b:/build/cpython-3.11.1/builddir/wasi# wasmtime run --mapdir /usr/local/lib::usr/local/lib python.wasm
Python 3.11.1 (main, Jan 16 2023, 20:30:30) [Clang 14.0.4 (https://github.com/llvm/llvm-project 29f1039a7285a5c3a9c353d05414 on wasi
Type "help", "copyright", "credits" or "license" for more information.
>>> 

The build is a bit akward since it copies zlib into the wasi-sdk sysroot, which I did because the wasi-env script seems to override any other place it could be discovered from:

cpython/Tools/wasm/wasi-env

Lines 63 to 64 in 1de4395

	PKG_CONFIG_PATH=""
	PKG_CONFIG_LIBDIR="${WASI_SYSROOT}/lib/pkgconfig:${WASI_SYSROOT}/share/pkgconfig"

I'd be open to contributing something to get this working by default for the wasi build, but couldn't find where the buildbot containers are defined and wanted to get some feedback on approaches before starting to modify things in the Tools/Wasm dir.

[brettcannon]

I believe the current buildbots are using https://github.com/tiran/ansible-wasm-buildbot to set them up.

[gpshead]

FYI this is probably the only issue blocking me making zlib required per https://discuss.python.org/t/lets-make-zlib-required-rather-than-optional-to-build-cpython/23062 and #91246

[kushaldas]

I can verify that static linking (of zlib) works as mentioned by @robbertvanginkel in #93819 (comment).

[brettcannon]

https://github.com/singlestore-labs/python-wasi also builds zlib, so there might be some inspiration there.

[brettcannon]

https://discord.com/channels/453584038356058112/915046161126137856/1070023227667124365 from Singlestore Labs suggests:

For the most part, I would modify the Makefiles to add --target=wasm32-unknown-wasi to the compiler flags. I would also add wasix to the compiler include flags and linker flags to fill the gaps between Wasi and POSIX. https://github.com/singlestore-labs/wasix

I know Christian did a bunch of work to eliminate the WASIX dependency from our builds.

[gzurl]

Folks, we are already offering a zlib build for WASI here as part of the WebAssembly Language Runtimes project. Let us know if this fulfills your needs.

[brettcannon]

@gzurl is there something you can upstream for this?

[gzurl]

@brettcannon, yes, we can take a look at this 👍 (most likely after CNCF Wasm Day)

[assambar]

@brettcannon can you give us some pointers as to where we could try contributing re-use of the prebuilt zlib that we offer in WLR? I was thinking about modifying https://github.com/python/cpython/blob/main/Tools/wasm/wasm_build.py with some code that can download and setup zlib (and why not more) automatically during the build. However, will you be willing to depend on WLR releases (https://github.com/vmware-labs/webassembly-language-runtimes/releases?q=libs)? I don't know if there is a CPython policy for that.

@tiran, we could also add zlib or other libraries to your container images. Let me know if you think this is fitting.

As an example, here are the dependencies we use for some builtin Python modules (zlib, uuid, sqlite3, bzip2) - https://github.com/vmware-labs/webassembly-language-runtimes/blob/main/python/v3.11.3/wlr-build-deps.sh

The build on our side will download all of those in the same "sysroot" folder and then setup pkg-config for cross compiling with that sysroot (and specifically PKG_CONFIG_LIBDIR=${PKG_CONFIG_SYSROOT_DIR}/lib/wasm32-wasi/pkgconfig). Each library has it's own .pc file that we use for cross-compiling to wasm. This allows for linking to the wasm32-wasi static libs just as with a native build flow, as shown in these examples - https://github.com/vmware-labs/webassembly-language-runtimes/tree/main/libs/examples

[brettcannon]

will you be willing to depend on WLR releases

Unfortunately I would rather not. We try to keep CPython has dependency-free as possible.

I honestly don't know why it's turned off to begin with since I would assume zlib is easy to compile.

[brettcannon]

Thanks to @katharosada in #91246 (comment) , we have verified that building zlib is just due to a lack of source, not an inherent issue with WebAssembly.

I'm closing this in favour of #91246 which would make zlib a required part of CPython and take care of the source availability problem.