Problems building on Python 3.9 related to Distutils

[hoffmang9]

I'm attempting to build wheels on python 3.7, 3.8, and now 3.9.

With recent changes in setuptools a couple of weeks ago I was forced to add:
CIBW_ENVIRONMENT_WINDOWS: SETUPTOOLS_USE_DISTUTILS=stdlib

To have the Windows MSVC build for 3.8 correctly put quotes around LIBPATH entries in the link command line. This entry is showing it improperly unquoted:
/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.27.29110\ATLMFC\lib\x64

Without the quotes it's then trying to link from C:\Program Files\ instead of the actual lib directories.

My new attempt to build with python 3.9 gave me this new error:
C:\cibw\python\python.3.9.0-rc1\tools\lib\site-packages\setuptools\distutils_patch.py:25: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first.
at - https://github.com/Chia-Network/chiapos/runs/1221477218?check_suite_focus=true#step:13:1697

The unquoted Windows link attempt is here: https://github.com/Chia-Network/chiapos/runs/1221477218?check_suite_focus=true#step:13:1883

Overall run can be found here - https://github.com/Chia-Network/chiapos/runs/1221477218?check_suite_focus=true

[henryiii]

Quick suggestion: limit or fix your setuptools version in pyproject.toml, say 42-49 in stress of just 42+. That should get you building again until this can be looked at properly (I'm on a phone currently).

[henryiii]

And 50.1 makes this the default again, for now. So maybe even pinning the latest version in pyproject.toml will fix things for you?

[Czaki]

To choose basic package versions use CIBW_DEPENDENCY_VERSIONS https://cibuildwheel.readthedocs.io/en/stable/options/#dependency-versions, not pyproject.toml

[hoffmang9]

As hinted, pyproject.toml changes didn't constrain things.

I was able to add a constraints.txt and hold setuptools to 50.1.0 - but python 3.9 still failed to create the correct link command line: https://github.com/Chia-Network/chiapos/runs/1222784576?check_suite_focus=true#step:13:1887

Huge thanks for the interim work around ideas. I'm here to test anything I can that would help.

[henryiii]

Odd, pyproject.toml should override anything else normally - that is the whole point of pyproject.toml, to enable completely reproducible builds, in CI (like cibuildwheel) as well as locally. The constrains versions should only be the starting points or if pyproject.toml is not present (unless manually disabled).

[hoffmang9]

This patch - Chia-Network/chiapos@24ee8b7

Still gave me setuptools 50.3 in this run: https://github.com/Chia-Network/chiapos/runs/1222660457#step:13:1720

[henryiii]

Internally, I believe pyproject.toml is respected. You start with the constrained version (and you see that in the log), but in the build process, you get a new version following pyproject.toml in a virtual environment - that's how PEP 517/518 builds work. Add the verbose flag to see how it works. Again, that's the whole point of PEP 518; you should be able to pin and require the exact build dependencies anywhere in a single place.

Can you add verbose and print out the setuptools.__version__ in your setup.py? That should match your pyproject.toml requirements.

[henryiii]

Yes, the line you point at is before the build isolation step; that's the surrounding environment, not the build environment.

[henryiii]

How does the lib error you are seeing compare to what you get with the "vendored" distutils? The vendored distutils comes from Python 3.9, so I'd rather expect SETUPTOOLS_USE_DISTUTILS=stdlib and SETUPTOOLS_USE_DISTUTILS=local to be very similar on Python 3.9.

Also, I would pin the latest setuptools in both pyproject.toml and in constraints.txt, just to be sure, for now. I would expect the latest version to be the best at building for Python 3.9 (though it looks like they've been talking about 3.9 occasionally in the changelog since around 45).

[hoffmang9]

When you had had said build environment I had cibuildwheel's docker in mind. With that "duh" out of the way, I can confirm that I'm getting 49.6, 50.1, or 50.3 as specified in pyproject.toml.

This is my current failure matrix. F for fail, ? for no data yet:

Setuptools version	49.6	50.1	50.2	50.3
DISTUTILS=stdlib	F	F	F	F
DISTUTILS=local	?	F	F	F

[henryiii]

The failure is only on 3.9 in all those F's? Or does it ever fail earlier?

By the way, pybind11 2.6 (not yet released) will provide tools to really clean up your setup.py if you want to. Also, you shouldn't need setup_requires and pyproject.toml, and I don't think setup_requires even works if you don't have it also listed in install_requires (at least, it does something, but doesn't do what you'd expect, based on docs and some testing, but I could be wrong there).

[hoffmang9]

I scoped down to just 3.9 to see if there was a possible success there and there is not. I'm attempting to read between the lines on this - https://github.com/pypa/setuptools/blob/master/CHANGES.rst

There is some cruft that's just wrong in setup.py but our Windows builds - especially - skips it.

I saw the beta release notification on pybind so already keeping an eye on that.

[henryiii]

Can you try building this directly on 3.9 on Windows without cibuildwheel? (I don't have a Windows machine setup). It looks like a pretty basic build, not seeing what you would do that would muddle with this, but 3.9 shipping with a broken distutils (and having it copied into setuptools) seems unlikely too.

[henryiii]

You can just add cmake to your pyproject.toml and drop all the custom code loading CMake for cibuildwheel, by the way. (not relevant here, since you are using setuptools on Windows)

[Czaki]

@hoffmang9 Did you have option to try build on windows machine without using cibuildwheel? Its may be python 3.9 related bug.

[henryiii]

Feel free to try a direct build using GitHub actions, using actions/setup-python@v2 you can already specify 3.9.

[hoffmang9]

My colleague @wjblanke had the python 3.9 build fail directly on Windows 10. Hints about the best repo to bug report? pypa/setuptools?

[Czaki]

I think it is best repository.

[henryiii]

I'd post it on setuptools, and include a cross reference pointing here.

[hoffmang9]

This issue was tracked down so closing this Chia-Network/chiapos#122