Wrong platform tag for wheel build on macOS 11 with universal2 installer

[ronaldoussoren]

Using a universal2 build of python 3.9.1rc1 on macOS with deployment target 10.9 I get a wheel with platform tag "macosx_11_0_universal2" instead of "macosx_10_9_universal2" as I had expected.

Sysconfig.get_platform() reports:

>>> import sysconfig
>>> sysconfig.get_platform()
'macosx-10.9-universal2'

Python setup.py bdist_wheel creates a file named pyobjc_core-7.0.1-cp39-cp39-macosx_11_0_universal2.whl.

$ pip list  | grep wheel
wheel             0.35.1

The python build I use is the experimental universal2 for 3.9.1rc1 on www.python.org.

Does wheel check the deployment target in the Mach-O header? If so, there might be a problem with that build (which I'll look into).

[ronaldoussoren]

I've done some more checking. I guess wheel checks the deployment target in the wheel. That's not reliable in this case.

If I build a fat binary for arm64 and x86_64 with deployment target 10.9 like this:

MACOSX_DEPLOYMENT_TARGET=10.9 cc -arch arm64 -arch x86_64 -o t t.m

... then "otool -vl" reports that the min-os-version is 11.0, even though the x86_64 branch of the fat binary works fine on macOS 10.15.

[ronaldoussoren]

To expand on that:

arch -x86_64 otool -vl t reports 10.9 as the deployment target (as expected)
arch -arm64 otool -vl t reports 11.0 as the deployment target.

Wheel currently picks up the 11.0 target for the platform tag in the wheel. It would be better to pick the lowest version.

I can provide a PR, but not right now.

Note that the current plan for the installers on python.org is to have an experimental universal2 installer with deployment target 10.9 for 3.9 (next to the current x86_64 installer), and make the univeral2 variant the default for 3.10.

BTW. Why does wheel look at the deployment target in the first place? Why not trust sysconfig.get_platform()?

[ronaldoussoren]

For my reference:

wheel/src/wheel/macosx_libfile.py

Line 279 in 39b6397

return max(versions_list)

That line selects the highest deployment target, which is suboptimal especially now that clang ensures that the deployment target for the arm64 architecture is at least 11.0.

[Czaki]

BTW. Why does wheel look at the deployment target in the first place? Why not trust sysconfig.get_platform()?

For example, to use c++17 you need macos in version at least 10.13. But when You install python from the official installer sys.get_platform() will return 10.9. And the wheel will not work on the system below 10.13.

There are also external libraries that could be compiled against a newer version of the system.
Or build system for package call cmake which internally set higher deployment target.

When the decision was made there is no public knowledge about ARM processor plan.

That line selects the highest deployment target, which is suboptimal especially now that clang ensures that the deployment target for the arm64 architecture is at least 11.0.

A month ago there is no public access to the new Apple ideas.
This decision was conservative. It is better to limit the system than upload on pypi wheel which will not work on a system which is suggested by the platform tag.

There was an option for multiple platform tag for macOS wheel but as I understand its support was removed. Maybe there should be added some additional logic for ARM.

But if then there should be a check if ARM minimal version is 11. Otherwise, it cannot be something like 10.9 because it will suggest that it will be working on 11.0 ARM

[Czaki]

Which macOS hardware is needed to build a universal wheel?

[ronaldoussoren]

Oh, that's annoying. I though sysconfig.get_platform() would get adjusted when building for a specific deployment target, but it doesn't. I'm not entirely convinced that the Mach-O reader is a good idea, but that's a discussion for another time.

Writing the patches for macOS 11 and arm64 for CPython took more time than I had expected at first (because, well, ...). My initial plan was to have full support for these in CPython releases and packaging tooling before the new arm64 were released, but that didn't go as expected. Still, we're pretty close.

I didn't know that wheel checked the Mach-O header, that's why I only now noticed this problem. Otherwise I'd have provided a PR earlier.

I'd prefer to use the minimal deployment target in the wheel platform tag. That's less conservative, but makes it possible to build wheels supporting the universal2 installer. Other than arm64 there's no real use case for specifying different deployment targets for the different architectures.

Checking the deployment target for the arm64 arm shouldn't be necessary: the current compiler will not set that to a version before 11.0, and even if it did the binary would work fine on macOS 11.

Note that the deployment target is not a guarantee that the binary will actually work, the binary can still try to use APIs introduced in a later version of the OS. But there's nothing you can do about that on the wheel side.

My current plan is to create a PR during the weekend (although I might be late, I'll also be working on back porting macOS11/arm64 support to CPython 3.8).

[ronaldoussoren]

Which macOS hardware is needed to build a universal wheel?

With the current release of CPython 3.9.1rc1 you can build a universal2 wheel on a mac system running macOS 11 (with either CPU type) and Xcode 12[1]. Xcode 12 itself can create universal2 binaries on macOS 10.15 as well, but I haven't had time to tweak _osx_support.py in the stdlib yet to detect if the compiler installed on 10.15 is good enough to build universal2 binaries.

[1] Xcode 12 or the "Command Line Tools" package for Xcode 12.

[Czaki]

Checking the deployment target for the arm64 arm shouldn't be necessary: the current compiler will not set that to a version before 11.0, and even if it did the binary would work fine on macOS 11.

I cannot agree. As I understand @agronholm is not familiar with macOS specific problems. And the problem of the targeting version will come back with MacOS 12 release. I have macOS hardware from university and have a plan to finish my Ph.D. so I may lose access to macOS hardware. Maybe best option is to finish the first PR fo fix the current problem and start next PR for solution which should work with versions like 12, 13 etc.

I'd prefer to use the minimal deployment target in the wheel platform tag. That's less conservative, but makes it possible to build wheels supporting the universal2 installer. Other than arm64 there's no real use case for specifying different deployment targets for the different architectures.

So if there is one library that needs 10.9 and all other files need 10.13 it should have 10.9 in the platform tag. Even if it will not work on older systems? I meet such package and it creates a big mess. It could be installed on an older system but only from the source. Which was hard to explain for an inexperienced user (he expect that if pip grab wheel it will work).

I rather prefer to build two wheels. one universal for systems 11.0 and one x86_64 for 10.9.

I'm a contributor to cibuildwheel package. Could we ask you for help with adding build of universal (or arm) wheel for macOS on ci infrastructure) pypa/cibuildwheel#473

[ronaldoussoren]

There will be a "univeral2" build of CPython, which will work on macOS 10.9 and later. For arm64 this will require macOS 11, but that's because Apple doesn't ship macOS 10.x for arm64.

To try to meet in the middle: Let's check for the deployment target per architecture and use the x86_64 deployment target in the platform tag, iff the arm64 deployment target is 11.0. We can reconsider this next year for macOS 12.

This way we can have "macosx_10_9_universal2" wheels that work on all systems running macOS 10.9 or later.

I've subscribed to the cibuildwheel issue. It is a program that I wanted to look into anyway to automate buildings wheels for a number of projects.

[Czaki]

I've subscribed to the cibuildwheel issue. It is a program that I wanted to look into anyway to automate buildings wheels for a number of projects.

Your welcome. My contribution to wheel comes from the needs of cibuildwheel.

[ronaldoussoren]

I've created a PR for this issue. With the PR the platform tag for the wheel for a "universal2" build will be calculated using the deployment target in the x86_64 part of the fat binary, iff the arm64 part contains deployment target "11.0.0".

This will allow creating wheels that target macOS 10.9 or later (in particular targeting the "universal2" variant of the installer on Python.org, introduced for Python 3.9.1).