propagate abiflags to wheel name on Windows #2325

davidhewitt · 2024-11-26T21:35:40Z

At the moment the windows builds are hard coded to use the none abi tag (unless they are abi3). I think this is probably technically incorrect, though in practice has worked fine historically because there was only one build on Windows... until 3.13t has come along with free-threading as a second option :)

It's important that we do emit proper abi tags (e.g. cp313, cp313t) because otherwise the two 3.13 wheels end up with the same name and can't be uploaded to PyPI. I think I also saw some install failures on jiter CI where a Python 3.13 build ended up installing the other build's wheel (I think pip detected an incorrect lib name and rejected it).

I suspect I might need to fix some tests...

davidhewitt · 2024-11-26T21:41:30Z

For what it's worth, I verified with pip debug --verbose (on a Windows machine) that the cp313 abi tags are accepted.

Also, numpy ships windows wheels with proper abi tags, e.g.

https://pypi.org/project/numpy/#numpy-2.1.3-cp313-cp313-win_amd64.whl
https://pypi.org/project/numpy/#numpy-2.1.3-cp313-cp313t-win_amd64.whl

davidhewitt · 2024-11-26T22:05:10Z

@ngoldbaum any idea about the 3.13t failure? Looks like a setup-python issue, I also just hit the same thing in pydantic/jiter#172

ngoldbaum · 2024-11-26T22:46:12Z

I reported it over at Quansight-Labs/setup-python#4. I honestly have no idea what changed on the runners.

ngoldbaum · 2024-11-26T23:01:18Z

never mind, the python.org ftp fell over: python/pythondotorg#2663

ngoldbaum · 2024-11-27T00:36:27Z

I also hit this over on rpds-py: https://github.com/ngoldbaum/rpds/actions/runs/12041384455/job/33573211395#step:4:137

The issue with python.org is fixed so I think if you restart CI it will at least get past installing python.

davidhewitt · 2024-11-27T10:29:41Z

I pushed an additional commit which should hopefully fix the test failures. I was looking in sysconfig on Windows and I noticed that lowercase variable "abiflags" is present on all platforms for 3.7+, whereas uppercase "ABIFLAGS" is only present on unix.

The two seem to always be equivalent on unix platforms, so I think we're fine to switch to lower-case "abiflags" as the thing we query out of the Python interpreter.

Result of grepping sysconfig directory

$ grep -Ri abiflags sysconfig 
sysconfig/cpython-freebsd-13.1-armv7.txt:       ABIFLAGS = ""
sysconfig/cpython-freebsd-13.1-armv7.txt:       abiflags = ""
sysconfig/pypy-macos-3.7-7.3.txt:       abiflags = ""
sysconfig/pypy-linux-3.7-7.3.txt:       abiflags = ""
sysconfig/cpython-omnios-3.9.txt:        ABIFLAGS = ""
sysconfig/cpython-omnios-3.9.txt:        abiflags = ""
sysconfig/cpython-win-3.7.txt:  abiflags = ""
sysconfig/cpython-openbsd-3.8-aarch64.txt:      ABIFLAGS = ""
sysconfig/cpython-openbsd-3.8-aarch64.txt:      abiflags = ""
sysconfig/pypy-linux-s390x-3.7-7.3.txt: ABIFLAGS = ""
sysconfig/pypy-linux-s390x-3.7-7.3.txt: abiflags = ""
sysconfig/cpython-linux-mips64-3.11.txt:        ABIFLAGS = ""
sysconfig/cpython-linux-mips64-3.11.txt:        abiflags = ""
sysconfig/cpython-freebsd-13.0-aarch64.txt:        ABIFLAGS = "m"
sysconfig/cpython-freebsd-13.0-aarch64.txt:        abiflags = "m"
sysconfig/cpython-macos-3.9-arm64.txt:  ABIFLAGS = ""
sysconfig/cpython-macos-3.9-arm64.txt:  abiflags = ""
sysconfig/cpython-aix-3.9.txt:  ABIFLAGS = ""
sysconfig/cpython-aix-3.9.txt:  abiflags = ""
sysconfig/cpython-linux-3.9.txt:        ABIFLAGS = ""
sysconfig/cpython-linux-3.9.txt:        abiflags = ""
sysconfig/pypy-linux-ppc64le-3.8-7.3.txt:       ABIFLAGS = ""
sysconfig/pypy-linux-ppc64le-3.8-7.3.txt:       abiflags = ""
sysconfig/cpython-linux-3.8.txt:        ABIFLAGS = ""
sysconfig/cpython-linux-3.8.txt:        abiflags = ""
sysconfig/pypy-linux-ppc64le-3.9-7.3.txt:       ABIFLAGS = ""
sysconfig/pypy-linux-ppc64le-3.9-7.3.txt:       abiflags = ""
sysconfig/cpython-openbsd-3.8.txt:      ABIFLAGS = ""
sysconfig/cpython-openbsd-3.8.txt:      abiflags = ""
sysconfig/cpython-haiku-3.9.txt:        ABIFLAGS = ""
sysconfig/cpython-haiku-3.9.txt:        abiflags = ""
sysconfig/cpython-gnu-3.12.txt: ABIFLAGS = ""
sysconfig/cpython-gnu-3.12.txt: abiflags = ""
sysconfig/cpython-win-mingw64-3.9.txt:        ABIFLAGS = ""
sysconfig/cpython-win-mingw64-3.9.txt:        abiflags = ""
sysconfig/graalpy-macos-3.8-arm64.txt:  ABIFLAGS = ""
sysconfig/graalpy-macos-3.8-arm64.txt:  abiflags = ""
sysconfig/pypy-linux-s390x-3.8-7.3.txt: ABIFLAGS = ""
sysconfig/pypy-linux-s390x-3.8-7.3.txt: abiflags = ""
sysconfig/cpython-freebsd-13.0-powerpc64.txt:   ABIFLAGS = ""
sysconfig/cpython-freebsd-13.0-powerpc64.txt:   abiflags = ""
sysconfig/nogil-linux-3.9.txt:  ABIFLAGS = ""
sysconfig/nogil-linux-3.9.txt:  abiflags = ""
sysconfig/cpython-wasi-3.12.txt:        ABIFLAGS = ""
sysconfig/cpython-wasi-3.12.txt:        abiflags = ""
sysconfig/cpython-freebsd-14.0-riscv64.txt:        ABIFLAGS = ""
sysconfig/cpython-freebsd-14.0-riscv64.txt:        abiflags = ""
sysconfig/pypy-windows-3.7-7.3.txt:        abiflags = ""
sysconfig/cpython-linux-mips64el-3.10.txt:      ABIFLAGS = ""
sysconfig/cpython-linux-mips64el-3.10.txt:      abiflags = ""
sysconfig/cpython-freebsd-13.0-powerpc64le.txt: ABIFLAGS = ""
sysconfig/cpython-freebsd-13.0-powerpc64le.txt: abiflags = ""
sysconfig/pypy-linux-s390x-3.9-7.3.txt: ABIFLAGS = ""
sysconfig/pypy-linux-s390x-3.9-7.3.txt: abiflags = ""
sysconfig/cpython-android-3.8-aarch64.txt:      ABIFLAGS = ""
sysconfig/cpython-android-3.8-aarch64.txt:      abiflags = ""
sysconfig/cpython-linux-3.7.txt:        ABIFLAGS = "m"
sysconfig/cpython-linux-3.7.txt:        abiflags = "m"
sysconfig/cpython-linux-3.8-aarch64.txt:    ABIFLAGS = ""
sysconfig/cpython-linux-3.8-aarch64.txt:    abiflags = ""
sysconfig/cpython-win-3.9-aarch64.txt:        abiflags = ""
sysconfig/cpython-dragonfly-3.8.txt:    ABIFLAGS = ""
sysconfig/cpython-dragonfly-3.8.txt:    abiflags = ""
sysconfig/cpython-freebsd-11.2.txt:     ABIFLAGS = "m"
sysconfig/cpython-freebsd-11.2.txt:     abiflags = "m"
sysconfig/pypy-linux-ppc64le-3.7-7.3.txt:       ABIFLAGS = ""
sysconfig/pypy-linux-ppc64le-3.7-7.3.txt:       abiflags = ""
sysconfig/cpython-netbsd-3.9.txt:       ABIFLAGS = ""
sysconfig/cpython-netbsd-3.9.txt:       abiflags = ""
sysconfig/pyston-3.8.txt:       ABIFLAGS = ""
sysconfig/pyston-3.8.txt:       abiflags = ""

davidhewitt · 2024-11-27T10:52:59Z

Ok will debug more later...

davidhewitt · 2024-11-27T21:30:56Z

Ok, I take it back. The abiflags seems to always be empty on windows, I missed that earlier.

sysconfig/cpython-win-3.7.txt:  abiflags = ""

... I've pushed a commit which manually encodes the expected flags.

messense · 2024-11-28T13:40:39Z

Seems like the wheel tag is still wrong for abi3 on Windows when building for 3.13t:

D:\a\_temp\54c3be4a-579c-4bd8-b031-a116c3fc0582\maturin.exe build --target x86_64-pc-windows-msvc --release --out dist -i python3.13t
🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.8
🐍 Not using a specific python interpreter (Automatically generating windows import library)
📡 Using build options features from pyproject.toml
   Compiling pyo3-build-config v0.23.2
   Compiling pyo3-ffi v0.23.2
   Compiling pyo3-macros-backend v0.23.2
   Compiling pyo3 v0.23.2
   Compiling pyo3-macros v0.23.2
   Compiling nh3 v0.2.18 (D:\a\nh3\nh3)
    Finished `release` profile [optimized] target(s) in 11.97s
📖 Found type stub file at nh3.pyi
📦 Built wheel for abi3 Python ≥ 3.8 to dist\nh3-0.2.18-cp38-abi3-win_amd64.whl

https://github.com/messense/nh3/actions/runs/12069973856/job/33658408634#step:6:32

davidhewitt · 2024-11-28T13:54:42Z

Agh thanks for flagging, I can try to investigate tonight if you don't get there first :)

messense · 2024-11-28T14:43:45Z

So the problem is that windows abi3 wheel currently doesn't not resolve python interpreter correctly for 3.13t, see

maturin/src/build_options.rs

Lines 372 to 391 in 57c74b6

    
           if env::var_os("PYO3_CROSS_LIB_DIR").is_some() { 
        
               // PYO3_CROSS_LIB_DIR should point to the `libs` directory inside base_prefix 
        
               // when cross compiling, so we fake a python interpreter matching it 
        
               eprintln!("⚠️  Cross-compiling is poorly supported"); 
        
               Ok(vec![PythonInterpreter { 
        
                   config: InterpreterConfig { 
        
                       major: *major as usize, 
        
                       minor: *minor as usize, 
        
                       interpreter_kind: InterpreterKind::CPython, 
        
                       abiflags: "".to_string(), 
        
                       ext_suffix: ".pyd".to_string(), 
        
                       pointer_width: None, 
        
                       gil_disabled: false, 
        
                   }, 
        
                   executable: PathBuf::new(), 
        
                   platform: None, 
        
                   runnable: false, 
        
                   implementation_name: "cpython".to_string(), 
        
                   soabi: None, 
        
               }])

and

maturin/src/build_options.rs

Lines 401 to 418 in 57c74b6

    
           eprintln!("🐍 Not using a specific python interpreter (Automatically generating windows import library)"); 
        
           // fake a python interpreter 
        
           Ok(vec![PythonInterpreter { 
        
               config: InterpreterConfig { 
        
                   major: *major as usize, 
        
                   minor: *minor as usize, 
        
                   interpreter_kind: InterpreterKind::CPython, 
        
                   abiflags: "".to_string(), 
        
                   ext_suffix: ".pyd".to_string(), 
        
                   pointer_width: None, 
        
                   gil_disabled: false, 
        
               }, 
        
               executable: PathBuf::new(), 
        
               platform: None, 
        
               runnable: false, 
        
               implementation_name: "cpython".to_string(), 
        
               soabi: None, 
        
           }])

these hacks work fine before but fell short with 3.13t.

I'm off to sleep now, feel free to investigate further.

This is an attempt to fix #2325 (comment) (Not tested yet, I wanted to push and see how this runs on CI as well as test locally.) --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

propagate abiflags to wheel name on Windows

a89372b

davidhewitt mentioned this pull request Nov 26, 2024

ci: remove workaround for free-threaded build w. maturin pydantic/jiter#172

Draft

add "m" abiflag on Windows 3.7

995e6ad

simplify fun_with_abiflags

551a505

manually define abiflags for windows

6f18204

ensure not panic

4f6a855

messense approved these changes Nov 28, 2024

View reviewed changes

messense merged commit d7f11c9 into PyO3:main Nov 28, 2024
38 checks passed

davidhewitt mentioned this pull request Nov 28, 2024

fix abi3 interpreter discovery on Windows #2333

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

propagate abiflags to wheel name on Windows #2325

propagate abiflags to wheel name on Windows #2325

davidhewitt commented Nov 26, 2024

davidhewitt commented Nov 26, 2024 •

edited

Loading

davidhewitt commented Nov 26, 2024

ngoldbaum commented Nov 26, 2024

ngoldbaum commented Nov 26, 2024

ngoldbaum commented Nov 27, 2024

davidhewitt commented Nov 27, 2024 •

edited

Loading

davidhewitt commented Nov 27, 2024

davidhewitt commented Nov 27, 2024

messense commented Nov 28, 2024

davidhewitt commented Nov 28, 2024

messense commented Nov 28, 2024

propagate abiflags to wheel name on Windows #2325

propagate abiflags to wheel name on Windows #2325

Conversation

davidhewitt commented Nov 26, 2024

davidhewitt commented Nov 26, 2024 • edited Loading

davidhewitt commented Nov 26, 2024

ngoldbaum commented Nov 26, 2024

ngoldbaum commented Nov 26, 2024

ngoldbaum commented Nov 27, 2024

davidhewitt commented Nov 27, 2024 • edited Loading

davidhewitt commented Nov 27, 2024

davidhewitt commented Nov 27, 2024

messense commented Nov 28, 2024

davidhewitt commented Nov 28, 2024

messense commented Nov 28, 2024

davidhewitt commented Nov 26, 2024 •

edited

Loading

davidhewitt commented Nov 27, 2024 •

edited

Loading