Use zlib-ng instead of zlib #8500

nulano · 2024-10-25T19:20:37Z

Alternative to #8495 .

Changes proposed in this pull request:

Replace zlib with zlib-ng built with CMake in Windows builds. Building our own zlib-ng allows us to use the latest release since the zlib-ng 2.2.2 release is missing the x86_64 Windows build.

Add a flag to PIL.features for zlib-ng. python3 -m PIL.report now looks like this:

--------------------------------------------------------------------
Pillow 11.1.0.dev0
Python 3.11.9 (tags/v3.11.9:de54cf5, Apr  2 2024, 10:12:12) [MSC v.1938 64 bit (AMD64)]
--------------------------------------------------------------------
Python executable is C:\hostedtoolcache\windows\Python\3.11.9\x64\python.exe
System Python files loaded from C:\hostedtoolcache\windows\Python\3.11.9\x64
--------------------------------------------------------------------
Python Pillow modules loaded from C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\PIL
Binary Pillow modules loaded from C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\site-packages\PIL
--------------------------------------------------------------------
--- PIL CORE support ok, compiled for 11.1.0.dev0
--- TKINTER support ok, loaded 8.6
--- FREETYPE2 support ok, loaded 2.13.3
--- LITTLECMS2 support ok, loaded 2.16
--- WEBP support ok, loaded 1.4.0
--- JPEG support ok, compiled for libjpeg-turbo 3.0.4
--- OPENJPEG (JPEG2000) support ok, loaded 2.5.2
--- ZLIB (PNG/ZIP) support ok, loaded 1.3.1.zlib-ng, compiled for zlib-ng 2.2.2
--- LIBTIFF support ok, loaded 4.6.0
--- RAQM (Bidirectional Text) support ok, loaded 0.10.1, fribidi 1.0.16, harfbuzz 10.0.1
--- LIBIMAGEQUANT (Quantization method) support ok, loaded 2.16.0
*** XCB (X protocol) support not installed
--------------------------------------------------------------------

I took inspiration from #8495 (comment) to make a benchmark with multiple compression values. It seems that zlib-ng usually produces a slightly larger file but in much less time during compression.

Script and results (click to expand)

import io
import os
import timeit

from PIL import Image

PATH = "Tests/images/hopper.png"  # or "Tests/images/effect_spread.png"
REPEAT = 1000

print(f"{Image.__version__ = }")
print(f"{Image.core.zlib_version = }")
print(f"Testing {PATH = } with {REPEAT} repetitions")

with open(PATH, "rb") as f:
    data_in = f.read()

reader = io.BytesIO(data_in)
writer = io.BytesIO()

img = Image.open(reader)
img.load()


def read_png():
    reader.seek(0, os.SEEK_SET)
    return Image.open(reader)


def write_png(compress_level):
    writer.seek(0, os.SEEK_SET)
    writer.truncate(0)
    img.save(writer, "PNG", compress_level=compress_level)


t = timeit.timeit(read_png, number=REPEAT * 10)
print(f"read PNG: time = {t:f} (sec)")

for compress_level in range(0, 10):
    t = timeit.timeit(lambda: write_png(compress_level), number=REPEAT)
    print(
        f"write PNG: time = {t:f} (sec); size = {len(writer.getvalue())} bytes; {compress_level = }"
    )

Image.open(io.BytesIO(writer.getvalue())).show()

# Image.__version__ = '11.1.0.dev0'
# Image.core.zlib_version = '1.3.1'
# Testing PATH = 'Tests/images/hopper.png' with 1000 repetitions
# read PNG: time = 0.157310 (sec)
# write PNG: time = 0.376219 (sec); size = 49353 bytes; compress_level = 0
# write PNG: time = 0.945004 (sec); size = 32403 bytes; compress_level = 1
# write PNG: time = 1.046024 (sec); size = 32156 bytes; compress_level = 2
# write PNG: time = 1.238888 (sec); size = 31792 bytes; compress_level = 3
# write PNG: time = 1.549707 (sec); size = 30715 bytes; compress_level = 4
# write PNG: time = 1.881609 (sec); size = 30529 bytes; compress_level = 5
# write PNG: time = 2.527365 (sec); size = 30343 bytes; compress_level = 6
# write PNG: time = 3.075476 (sec); size = 30277 bytes; compress_level = 7
# write PNG: time = 5.036960 (sec); size = 30183 bytes; compress_level = 8
# write PNG: time = 6.598194 (sec); size = 30166 bytes; compress_level = 9

# Image.__version__ = '11.1.0.dev0'
# Image.core.zlib_version = '1.3.1.zlib-ng'
# Testing PATH = 'Tests/images/hopper.png' with 1000 repetitions
# read PNG: time = 0.178209 (sec)
# write PNG: time = 0.382191 (sec); size = 49353 bytes; compress_level = 0
# write PNG: time = 0.549274 (sec); size = 40354 bytes; compress_level = 1
# write PNG: time = 0.770737 (sec); size = 31458 bytes; compress_level = 2
# write PNG: time = 0.934707 (sec); size = 31096 bytes; compress_level = 3
# write PNG: time = 0.977891 (sec); size = 30836 bytes; compress_level = 4
# write PNG: time = 1.101530 (sec); size = 30771 bytes; compress_level = 5
# write PNG: time = 1.212539 (sec); size = 30649 bytes; compress_level = 6
# write PNG: time = 1.383462 (sec); size = 30217 bytes; compress_level = 7
# write PNG: time = 1.946067 (sec); size = 30175 bytes; compress_level = 8
# write PNG: time = 2.788035 (sec); size = 30166 bytes; compress_level = 9

# Image.__version__ = '11.1.0.dev0'
# Image.core.zlib_version = '1.3.1'
# Testing PATH = 'Tests/images/effect_spread.png' with 1000 repetitions
# read PNG: time = 0.096424 (sec)
# write PNG: time = 0.410247 (sec); size = 49353 bytes; compress_level = 0
# write PNG: time = 1.194126 (sec); size = 42199 bytes; compress_level = 1
# write PNG: time = 1.164452 (sec); size = 42096 bytes; compress_level = 2
# write PNG: time = 1.182063 (sec); size = 41977 bytes; compress_level = 3
# write PNG: time = 1.404397 (sec); size = 41011 bytes; compress_level = 4
# write PNG: time = 1.531614 (sec); size = 40969 bytes; compress_level = 5
# write PNG: time = 1.720916 (sec); size = 40903 bytes; compress_level = 6
# write PNG: time = 1.806385 (sec); size = 40896 bytes; compress_level = 7
# write PNG: time = 1.914409 (sec); size = 40889 bytes; compress_level = 8
# write PNG: time = 1.936970 (sec); size = 40889 bytes; compress_level = 9

# Image.__version__ = '11.1.0.dev0'
# Image.core.zlib_version = '1.3.1.zlib-ng'
# Testing PATH = 'Tests/images/effect_spread.png' with 1000 repetitions
# read PNG: time = 0.088179 (sec)
# write PNG: time = 0.412940 (sec); size = 49353 bytes; compress_level = 0
# write PNG: time = 0.607201 (sec); size = 48241 bytes; compress_level = 1
# write PNG: time = 0.852157 (sec); size = 41287 bytes; compress_level = 2
# write PNG: time = 1.026876 (sec); size = 41168 bytes; compress_level = 3
# write PNG: time = 1.044624 (sec); size = 41134 bytes; compress_level = 4
# write PNG: time = 1.200349 (sec); size = 41120 bytes; compress_level = 5
# write PNG: time = 1.214951 (sec); size = 41118 bytes; compress_level = 6
# write PNG: time = 0.983906 (sec); size = 40895 bytes; compress_level = 7
# write PNG: time = 0.989961 (sec); size = 40889 bytes; compress_level = 8
# write PNG: time = 1.540352 (sec); size = 40889 bytes; compress_level = 9

Looking at the build logs, zlib-ng seems to have been properly detected by webp, libtiff, libpng, and freetype.

TODO:

Run the test suite on Windows arm64. I can use a borrowed M2 MacBook, but I won't have access to it at least until Monday.
We should also update the Linux...
...and macOS builds for consistency if we decide that we do want to go with zlib-ng.

dofuuz · 2024-10-26T01:19:30Z

zlib-ng v2.2.2 x86_64-compat was not released because of CI test error. It'll be released at next version again.
So, downloading pre-built binary is still a valid way.

Anyway, building with CMake is also a fine solution. I'll follow Pillow devs' decision and close my PR #8495 when it's decided.

hugovk · 2024-10-30T14:21:03Z

Here's a chart of the write times for different compression levels and images:

hopper.png: 1.3.1 zlib-ng (red) is much better than 1.3.1 (blue)
effect_spread.png: 1.3.1 zlib-ng (green) is noticeably better than 1.3.1 (yellow)

Summary: zlib-ng is much faster.

And file sizes:

compression level 0: same for all
compression level 1: 1.3.1 zlib-ng is much worse than 1.3.1
compression level 2-3: 1.3.1 zlib-ng is slightly better than 1.3.1
compression level 4-6: 1.3.1 zlib-ng is even more slightly worse than 1.3.1
compression level 7-9: about the same, zlib-ng ever so slightly better than 1.3.1

Summary: zlib-ng has same file sizes for compression level 0 and bigger files for compression level 1, but if you care about size, any higher compression level gives more or less the same file size (and zlib-ng is faster).

nulano · 2024-11-02T12:34:43Z

I've now checked that the Windows arm64 wheels pass the test suite on an MacBook Pro M2 Max (tested Python versions: 3.9.10, 3.10.11, 3.11.6, 3.12.0, 3.13.0).

Benchmark results comparing the Python 3.13 wheels of the 11.0.0 release and the pull request build (click to expand)

Image.__version__ = '11.0.0'
Image.core.zlib_version = '1.3.1'
Testing PATH = '../Pillow-11.0.0/Pillow-11.0.0/Tests/images/hopper.png' with 1000 repetitions
read PNG: time = 0.589200 (sec)
write PNG: time = 0.555782 (sec); size = 49353 bytes; compress_level = 0
write PNG: time = 1.114117 (sec); size = 32403 bytes; compress_level = 1
write PNG: time = 1.198036 (sec); size = 32156 bytes; compress_level = 2
write PNG: time = 1.402062 (sec); size = 31792 bytes; compress_level = 3
write PNG: time = 1.758227 (sec); size = 30715 bytes; compress_level = 4
write PNG: time = 2.065863 (sec); size = 30529 bytes; compress_level = 5
write PNG: time = 2.774447 (sec); size = 30343 bytes; compress_level = 6
write PNG: time = 3.542967 (sec); size = 30277 bytes; compress_level = 7
write PNG: time = 5.532757 (sec); size = 30183 bytes; compress_level = 8
write PNG: time = 7.352626 (sec); size = 30166 bytes; compress_level = 9

Image.__version__ = '11.1.0.dev0'
Image.core.zlib_version = '1.3.1.zlib-ng'
Testing PATH = '../Pillow-11.0.0/Pillow-11.0.0/Tests/images/hopper.png' with 1000 repetitions
read PNG: time = 0.569872 (sec)
write PNG: time = 0.529697 (sec); size = 49353 bytes; compress_level = 0
write PNG: time = 0.723248 (sec); size = 40354 bytes; compress_level = 1
write PNG: time = 0.942190 (sec); size = 31458 bytes; compress_level = 2
write PNG: time = 1.009014 (sec); size = 31096 bytes; compress_level = 3
write PNG: time = 1.053823 (sec); size = 30836 bytes; compress_level = 4
write PNG: time = 1.071575 (sec); size = 30771 bytes; compress_level = 5
write PNG: time = 1.170958 (sec); size = 30649 bytes; compress_level = 6
write PNG: time = 1.503532 (sec); size = 30217 bytes; compress_level = 7
write PNG: time = 2.005649 (sec); size = 30175 bytes; compress_level = 8
write PNG: time = 2.626763 (sec); size = 30166 bytes; compress_level = 9

Image.__version__ = '11.0.0'
Image.core.zlib_version = '1.3.1'
Testing PATH = '../Pillow-11.0.0/Pillow-11.0.0/Tests/images/effect_spread.png' with 1000 repetitions
read PNG: time = 0.234989 (sec)
write PNG: time = 0.550197 (sec); size = 49353 bytes; compress_level = 0
write PNG: time = 1.305417 (sec); size = 42199 bytes; compress_level = 1
write PNG: time = 1.334536 (sec); size = 42096 bytes; compress_level = 2
write PNG: time = 1.384041 (sec); size = 41977 bytes; compress_level = 3
write PNG: time = 1.622062 (sec); size = 41011 bytes; compress_level = 4
write PNG: time = 1.721847 (sec); size = 40969 bytes; compress_level = 5
write PNG: time = 1.927218 (sec); size = 40903 bytes; compress_level = 6
write PNG: time = 2.054222 (sec); size = 40896 bytes; compress_level = 7
write PNG: time = 2.198446 (sec); size = 40889 bytes; compress_level = 8
write PNG: time = 2.199249 (sec); size = 40889 bytes; compress_level = 9

Image.__version__ = '11.1.0.dev0'
Image.core.zlib_version = '1.3.1.zlib-ng'
Testing PATH = '../Pillow-11.0.0/Pillow-11.0.0/Tests/images/effect_spread.png' with 1000 repetitions
read PNG: time = 0.240964 (sec)
write PNG: time = 0.538218 (sec); size = 49353 bytes; compress_level = 0
write PNG: time = 0.799972 (sec); size = 48241 bytes; compress_level = 1
write PNG: time = 1.091412 (sec); size = 41287 bytes; compress_level = 2
write PNG: time = 1.177194 (sec); size = 41168 bytes; compress_level = 3
write PNG: time = 1.191972 (sec); size = 41134 bytes; compress_level = 4
write PNG: time = 1.195040 (sec); size = 41120 bytes; compress_level = 5
write PNG: time = 1.190594 (sec); size = 41118 bytes; compress_level = 6
write PNG: time = 1.280668 (sec); size = 40895 bytes; compress_level = 7
write PNG: time = 1.293338 (sec); size = 40889 bytes; compress_level = 8
write PNG: time = 1.738161 (sec); size = 40889 bytes; compress_level = 9

hugovk · 2024-11-18T12:21:09Z

Similar trend for time and identical numbers for size:

radarhere · 2024-11-25T09:55:58Z

winbuild/build_prepare.py

+        "license": "LICENSE.md",
+        "patch": {
+            r"CMakeLists.txt": {
+                "set_target_properties(zlib PROPERTIES OUTPUT_NAME zlibstatic${{SUFFIX}})": "set_target_properties(zlib PROPERTIES OUTPUT_NAME zlib)",  # noqa: E501


If zlib-ng creates zlibstatic.lib by default, then wouldn't it seem helpful to detect that in setup.py, in case a user had built it themselves? nulano#44

winbuild: Replace zlib with zlib-ng built with CMake

3e0849b

nulano mentioned this pull request Oct 25, 2024

Replace Windows zlib with zlib-ng #8495

Closed

PIL.features: Add a compile-time zlib-ng feature flag and version number

7885066

nulano force-pushed the zlib-ng branch from 0a75993 to 7885066 Compare October 25, 2024 19:32

nulano added Build Windows Performance labels Oct 25, 2024

radarhere mentioned this pull request Oct 26, 2024

Only replace version suffix if zlib-ng is present nulano/Pillow#41

Merged

Merge branch 'main' into zlib-ng

c07e089

radarhere and others added 2 commits October 26, 2024 11:23

Only replace version suffix if zlib-ng is present

ed910a6

Merge branch 'main' into zlib-ng

5ab79bc

nulano mentioned this pull request Oct 28, 2024

Saving PNG images with PIL is 4 times slower than saving them with OpenCV #5986

Open

Merge branch 'main' into zlib-ng

3f5f3bb

radarhere mentioned this pull request Nov 20, 2024

Use zlib-ng in Linux wheels nulano/Pillow#42

Merged

Use zlib-ng on Linux

69c9a7f

nulano added the Linux label Nov 21, 2024

radarhere changed the title ~~Windows: Use zlib-ng instead of zlib~~ Use zlib-ng instead of zlib Nov 22, 2024

Merge branch 'main' into zlib-ng

dab795c

This was referenced Nov 22, 2024

Use zlib-ng on macOS nulano/Pillow#43

Open

Detect zlibstatic.lib when building for Windows nulano/Pillow#44

Open

radarhere reviewed Nov 25, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use zlib-ng instead of zlib #8500

Use zlib-ng instead of zlib #8500

nulano commented Oct 25, 2024 •

edited

Loading

dofuuz commented Oct 26, 2024 •

edited

Loading

hugovk commented Oct 30, 2024

nulano commented Nov 2, 2024 •

edited

Loading

hugovk commented Nov 18, 2024

radarhere Nov 25, 2024 •

edited

Loading

Use zlib-ng instead of zlib #8500

Are you sure you want to change the base?

Use zlib-ng instead of zlib #8500

Conversation

nulano commented Oct 25, 2024 • edited Loading

dofuuz commented Oct 26, 2024 • edited Loading

hugovk commented Oct 30, 2024

nulano commented Nov 2, 2024 • edited Loading

hugovk commented Nov 18, 2024

radarhere Nov 25, 2024 • edited Loading

Choose a reason for hiding this comment

nulano commented Oct 25, 2024 •

edited

Loading

dofuuz commented Oct 26, 2024 •

edited

Loading

nulano commented Nov 2, 2024 •

edited

Loading

radarhere Nov 25, 2024 •

edited

Loading