[WORKAROUND] Unable to build PyYAML < 6.0.1 from source or sdist

[nitzmahone]

In the instructions below, replace pip as needed with a pip invocation matching your target Python environment (eg, pip3, /usr/local/bin/python3.11 -m pip).

Pre-seed wheel cache (recommended)

This solution pre-seeds pip's wheel cache with a locally-built PyYAML wheel, accessible to any subsequent installation using the same pip cache.

# create a constraint file that limits the Cython version to one that should work
echo 'Cython < 3.0' > /tmp/constraint.txt

# seed pip's local wheel cache with a PyYAML wheel
PIP_CONSTRAINT=/tmp/constraint.txt pip wheel PyYAML==5.4.1

# install PyYAML itself, or any other package(s) that ask for the PyYAML version you just built
pip install 'PyYAML==5.4.1'

Inline constraint (simpler, more likely to break)

This solution globally constrains the Cython version for all packages being installed by this pip invocation (including nested/child package installs), which could break other packages installed at the same time.

# create a constraint file that limits the Cython version to one that should work
echo 'Cython < 3.0' > /tmp/constraint.txt

# install PyYAML itself (or other packages that need it); any package requiring Cython will be constrained to `Cython < 3.0`
PIP_CONSTRAINT=/tmp/constraint.txt pip install 'PyYAML==5.4.1'

Background

With the release of Cython 3, all older versions of PyYAML can no longer be installed from unmodified source or sdist (ie, where a wheel is unavailable for the platform and/or Python), since the Cython version was not capped to a working version for all older PyYAML releases. For various reasons, it is untenable to release new sdists/wheels for these old PyYAML versions with the new required Cython<3.0 build dependency constraint.

pip has mostly-undocumented support for "inherited" constraints at install-time by setting the PIP_CONSTRAINT environment variable (which is inherited by child build processes, unlike the CLI --constraint arg).

Sample error output from pip install

If you're seeing an error similar to this when installing a version of PyYAML older than 6.0.1, the preceding solutions may help.

...
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [48 lines of output]
      running egg_info
      writing lib/PyYAML.egg-info/PKG-INFO
      writing dependency_links to lib/PyYAML.egg-info/dependency_links.txt
      writing top-level names to lib/PyYAML.egg-info/top_level.txt
      Traceback (most recent call last):

...

          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]

[leiflinse-trivector]

In a case where 5.4.1 arrives as a dependency from requirements.txt the workaround for us looks like this. Unfortunately when we build just the wheel with PIP_CONSTRAINT, that wheel is not used when installing from requirements.txt.

echo 'Cython < 3.0' > /tmp/constraint.txt
PIP_CONSTRAINT=/tmp/constraint.txt pip install 'PyYAML==5.4.1'

pip install -r requirements.txt

Thanks for posting the workaround. It was very helpful.

[nitzmahone]

@leiflinse-trivector Interesting- we (Ansible) have successfully deployed this workaround extensively throughout our CI infra on dozens of different OSs from Python 2.7-3.12 and numerous pip versions, just as described above. One caveat I purposely omitted from the description above (since I assumed people were only building PyYAML for platforms/Pythons that didn't already have a wheel): if the index server pip consults has an "applicable" PyYAML wheel, it seems to prefer the wheel on the index server to the local one. This seems to happen more if the PyYAML requirement is unconstrained by the dependent package- the presence of any exact or upper-bound constraint on PyYAML seems to make the locally-cached wheel eligible for use (eg, if the dependent package asks for PyYAML==5.4.1 or PyYAML<6)- if the dependent package doesn't specify an upper bound and is incompatible with PyYAML 6.x, it's already arguably broken anyway, but in that case, you can also use the PIP_CONSTRAINT trick to augment the broken package/requirements install with your own constraint for eg, PyYAML<6 that should cause it to use the locally-cached wheel.

[reid-harrison]

Is there a similar workaround for pipenv? Honestly this is what I'm doing but not sure I like it...

pip3 install "cython<3.0" wheel && pip3 install --no-build-isolation "pyyaml==5.4.1" && \
pipenv sync --system 

[nitzmahone]

@reid-harrison No idea here- not a pipenv user. That said, all you really need is a locally-built 5.4.1 wheel that's been constrained as necessary, so why not just do the pip wheel as describe above to create the wheel, then just install the wheel that pip created in CWD with pipenv as normal?

[nanonyme]

Just a warning to maintainers: PyYAML supporting Python 3.13 is supposedly going to require switching to Cython 3.

[nitzmahone]

@nanonyme Yeah, I've been doing all my local 3.13 testing against Cython 3 with the old_build_ext stuff and it's been working fine. We'll probably end up implementing at least conditional support for that in the update that supports 3.13, since the "shred the extension build to ditch all the distutils-isms" thing is still in flux.

[Uzume]

@nanonyme [...] supporting Python 3.13 is supposedly going to require switching to Cython 3.

My understanding is this is due to python/cpython#104775 (CPython 3.13+) and cython/cython#5128 (Cython 0.29.36+).

@nitzmahone [...] the "shred the extension build to ditch all the distutils-isms" thing is still in flux.

Care to elucidate what is holding back the jettison of distutils APIs in #797 (but feel free to comment over there where it is probably more pertinent to the discussion)?