Troubleshooting Python 3.12 uWSGI wheel

[ipmb]

Description

We've been building wheels for uWSGI successfully for a while using cibuildwheel. The latest version has some conditional code branches in the C code that need to run when it is running with Python 3.12. The 3.12 wheels built with cibuildwheel are showing as being built against 3.7.17. Building from source or manually creating a wheel and installing it works as expected on 3.12.

It's not clear to me where the issue stems from, but I figured this would be a good place to ask. How can we get these wheels working on Python 3.12?

Some additional troubleshooting steps and conversation on the issue is here: unbit/uwsgi#2579 (comment)

Build log

https://github.com/lincolnloop/pyuwsgi-wheels/actions/runs/7144198124/job/19457305955

CI config

https://github.com/lincolnloop/pyuwsgi-wheels/actions/runs/7144198124/workflow

[webknjaz]

@ipmb wow, why do you build an sdist AFTER making the wheels?

[webknjaz]

Also, I see that you make that sdist multiple times. Really, it should be built once. And it's best to build all the wheels out of that one sdist.

[webknjaz]

@ipmb it does build cp312 wheels. It's visible in your log: https://github.com/lincolnloop/pyuwsgi-wheels/actions/runs/7144198124/job/19457305955#step:5:6994.
You have CIBW_SKIP: cp36-* pp* which means that you want to skip building CPython 3.6 and PyPy platform-specific wheels. But the default matrix is big — it makes a lot more wheels, unless you specify CIBW_BUILD to override that: https://cibuildwheel.readthedocs.io/en/stable/options/#build-skip.

[webknjaz]

Also, I see that you make that sdist multiple times. Really, it should be built once. And it's best to build all the wheels out of that one sdist.

Here's one of my recent examples where the CI builds an sdist first. It then builds a matrix of wheels out of that sdist. After that, it runs tests from the sdist installing those pre-built wheels (so no testing against a Git checkout). Finally, it uploads (in publish mode only) all those wheels and the sdist to PyPI. No Git checkout is needed in either of those post-sdist jobs!
See https://github.com/aio-libs/yarl/actions/runs/7162519528.

One of the reasons is that the downstream packagers treat sdists from PyPI as the source of truth. And run builds and tests from them. Pip also builds wheels from sdists when wheels aren't available so naturally, you'd want this to be functional (not only building from the Git checkout).

[henryiii]

FYI, you should have a pyproject.toml. Python 3.12 doesn't come with setuptools or distutils, so you should be declaring that you use them in pyproject.toml.

All Python versions are present in the manylinux container, and I'd guess that you are somehow picking up one via a search rather than using the current one, or including a directory that they are in above the correct one. Like most setup.py based builds, there's a lot of custom code so I can't immediately see the culprit, but that's my guess.

Also, I also couldn't see where the "Built against" was in the test printout in the logs. I'd recommend making sure that's visible in the CI (not just downloading wheels manually) before debugging it.

[ipmb]

Thank you for the feedback @webknjaz. I'll look into optimizing the process around how we're creating the sdist.

My issue is not that the wheels aren't built it's that there is code in uWSGI which identifies which version of Python it is running on. In the Python 3.12 wheels that are generated, the uWSGI appears to be built using Python 3.7.17.

[mayeut]

From unbit/uwsgi#2579 (comment), it does not seem isolated to python 3.12 (at least python 3.11 is also affected or so it seems).
As @henryiii said, there's a lot of custom code in setup.py so it's hard to tell what's going on.

Given it's not isolated to python 3.12 and build times are much longer on python 3.7 than the others, my feeling is that there's some build cache being setup on the first run (which uses python 3.7) that's being reused on following builds.
You should increase build verbosity to confirm whether this is the case or not. In any case, increasing build verbosity should help pinpoint what's going on.

[Lalufu]

To (hopefully) clarify a bit:
There's two usages of the term "python" here.

• The python binary used to run the wheel build process (python -m pip wheel ...)
• The python development headers picked up when the wheel build process produces a machine code binary instead of just python source

From the pipeline logs I've seen, the first is correct, when building a wheel for python 3.x, the build process calls python3.x -m pip wheel ....
It's the second that's causing issues here, the python development headers that get found and used by the build always seem to be the headers for python 3.7.17, independent of the version of the python binary used to run the build. uWSGI (for better or worse) integrates pretty deeply with the interpreter, and does slightly different things for different versions of python it is built against.
I cannot tell if this mismatch is because the development headers of 3.7.17 are the only ones present in the pipeline, or if the build process just fails to find the correct ones for the version of python that is being run.

[webknjaz]

@Lalufu note that the Linux wheels are made in containers. So you're kinda limited by what's available. And the initial build is made by running python -m pip wheel or python -m build --wheel (there's a setting). Then, those wheels are modified by auditwheel --repair which is what makes them standard-compliant.
cibuildwheel mostly just wraps the standard tools that people would otherwise use directly.
So you can try seeing if you can produce wheels you want in your local environment.
If that works, try running the same commands in the manylinux containers.
This should help you narrow down what works or doesn't and in what cases.
With that info, people here might be able to help out. While this doesn't seem to be directly related to the cibuildwheel project (because it's just a glue between many single-purpose tools), we'll see some mismatches in your configuration of other tools.

Oh, and how did you build it in the past? Any examples?

[ipmb]

It is running here with verbosity increased. Interesting to note that I'm seeing this output from everything after the 3.7 build:

    *** uWSGI compiling server core ***
    core/utils.o is up to date
    core/protocol.o is up to date
    core/socket.o is up to date
    ...

So @mayeut seems to be on the right track. It looks like it isn't starting with a clean environment before each build which is not what I expected. What can I do to ensure it starts each build without any artifacts from the previous one?

[mayeut]

What can I do to ensure it starts each build without any artifacts from the previous one?

This is uWSGI specific.
Reading https://github.com/unbit/uwsgi/blob/master/uwsgiconfig.py, it seems you can either call this file with --clean in before build or set the UWSGI_FORCE_REBUILD environment variable to 1.

[Lalufu]

`make -C uwsgi clean`, or (if you don't have `make`, `find uwsgi -name '*,o' -delete` should work. The object files hanging around between build would explain the effects.

[ipmb]

It looks like that worked. https://github.com/lincolnloop/pyuwsgi-wheels/actions/runs/7201594896

I'm confused why these artifacts would be present across builds. I thought cibuildwheel was copying the project to an isolated directory in the container and then building from there. Am I misunderstanding?

[henryiii]

It does not redo it if the image does not change. The surprised some when PyPy started using the same image and setuptools didn’t include PyPy vs CPython in the build cache names (they fixed that pretty quickly though).

[webknjaz]

FTR I just hit a case with Cython building the modules in-place and each new Python version wheel includes *.so files from the previous ones, accumulating more of them in the newer wheels.

It looks like this in the output:

  copying src/pylibsshext/sftp.cpython-36m-darwin.so -> build/lib.macosx-10.9-x86_64-cpython-37/pylibsshext
  copying src/pylibsshext/sftp.cpython-37m-darwin.so -> build/lib.macosx-10.9-x86_64-cpython-37/pylibsshext

or:

adding 'pylibsshext/session.cpython-310-x86_64-linux-gnu.so'
adding 'pylibsshext/session.cpython-311-x86_64-linux-gnu.so'
adding 'pylibsshext/session.cpython-312-x86_64-linux-gnu.so'
adding 'pylibsshext/session.cpython-36m-x86_64-linux-gnu.so'
adding 'pylibsshext/session.cpython-37m-x86_64-linux-gnu.so'
adding 'pylibsshext/session.cpython-38-x86_64-linux-gnu.so'
adding 'pylibsshext/session.cpython-39-x86_64-linux-gnu.so'
adding 'pylibsshext/session.pxd'
adding 'pylibsshext/session.pypy310-pp73-x86_64-linux-gnu.so'
adding 'pylibsshext/session.pypy37-pp73-x86_64-linux-gnu.so'
adding 'pylibsshext/session.pypy38-pp73-x86_64-linux-gnu.so'
adding 'pylibsshext/session.pypy39-pp73-x86_64-linux-gnu.so'
adding 'pylibsshext/session.pyx'

[joerick]

This was different for a few years when pip did 'out-of-tree' builds, but these days its the responsibility of the build backend to not get confused by old build artifacts in the source tree.

cibuildwheel can't automatically 'clean', because we don't know which files are generated vs. which are source. But if you need to work around an issue in your specific project, you can tidy up old build artifacts in CIBW_BEFORE_BUILD.