Non-windows wheels

[Suor]

I saw some issues regarding this, but opted to create a new sort of umbrella one. This is basically about all binary wheels.

We are using PyYAML in dvc, where we support all non-dead Python versions, e.g. 2.7, 3.5, 3.6, 3.7, 3.8. We might ignore 2.7 as it ends soon, but not the other ones. We need wheels because we try to make install as easy as possible, while still making it parse yaml fast.

Wheels for all Python/OS combination could be built and deployed using Travis, this is what we do and here are travis docs on this. Setting up travis will solve the wheels issue once and for all.

So what do you think about it? I don't want this to become an another +1 issue :). If this is what you are willing to add then we are willing to help/provide a PR.

[bsolomon1124]

Are there any issues preventing manylinux and macosx wheels being built from a technical perspective? If not, what is the opposition to this?

[nitzmahone]

No real technical hurdles, just the time to put it together and make it bulletproof. IME many users don't understand enough about Python/pip/etc to know how to deal with a bad wheel (made complex by the embedding of static libyaml + Cython ext), so if there's anything wrong, we'll be flooded with "your package is busted" issues. We have plenty of that already with people hitting problems on debian/ubuntu-hacked-pip and all the mess with Mac homebrew/pyenv Python. We really don't want more accusations of busted packaging- especially if they're legitimate (meaning it needs to be right the first time).

I'm the only current pyyaml maintainer with Python packaging experience, and I'm buried in Ansible projects most of the time. Getting the Windows wheel builds working was a pretty awful nightmare, with the various VC runtime versions and SDKs needed by the different Python versions installed side-by-side and hacked to work, but Linux is generally easier. I've prototyped manylinux1/manylinux2010 container-based wheel builds locally with success, and I don't imagine MacOS would be too difficult- just have to decide which version of the OS/XCode we want to build against and 🤞 that it doesn't have issues we have to work around (as they so often seem to). I don't recall if libyaml's static library builds on POSIX are functional, but that'd definitely be a pre-requisite for those wheels to cut down on potential issues. I know I had to do some work on the Windows static library builds to get them working again.

Last time I talked about it with the other maintainers, we're not comfortable doing continuous-delivery-style builds on pyyaml either. It's much less a technical issue than wanting a human gate on releases (since, eg, we smoke-test most new releases under the Ansible CI and other things), and also not putting PyPI creds/keys in a publicly-hosted CI (as there have been numerous cases of leakage and fallout). The Windows wheel builds currently store their artifacts on the AppVeyor job, then we download them and publish to PyPI. We could do the same for manylinuxX and MacOS on AppVeyor, but IIRC Travis would need an intermediate artifact storage to support that publishing workflow (which means baking more credentials in there). Azure Pipelines would also be an option, though last I checked they still had no way to interactively debug a pipeline build, which makes things more difficult if when there are problems.

So that's where we're at... If someone wants to hack together a PR for manylinux/manylinux2010 wheel builds and/or MacOS wheel builds on AppVeyor that store them as artifacts, I'd probably be willing to publish them as a beta and run them through as much CI as I can (eg Ansible's many CI distros), but I don't know when I'll have time to put it together myself.

[bsolomon1124]

Thank you for the thorough explanation @nitzmahone .

So that's where we're at... If someone wants to hack together a PR for manylinux/manylinux2010 wheel builds and/or MacOS wheel builds on AppVeyor that store them as artifacts.

I may very well give this a shot.

[perlpunk]

I recently switched to GitHub Actions for some of my perl modules, and it works nice, has workflows, artifacts, Linux, MacOS and Windows.
Maybe that would also be an option for PyYAML?

[bsolomon1124]

@perlpunk yes, GitHub Actions is quite nice in that aspect. AppVeyor is a Windows-centric thing, as far as I know. (Have not really worked with it.)

I see that PyYAML currently uses Travis. It's actually possible to get MacOS/Python working on Travis CI, but it's an unofficial way to do things since they don't officially support it.

[nitzmahone]

Moving the Windows stuff somewhere else would likely be very difficult until we stop building for anything older than Python 3.6. The Linux side just needs container support and artifact storage without credentials (which GHA does IIRC). The Mac side might or might not work there- with the XCode/Python changes that have been announced, I'm not sure if we'll have to build the wheels on an older MacOS version to maintain compatibility (GHA only provides 10.15 right now, where other services provide several options).

[bsolomon1124]

I'm not sure if we'll have to build the wheels on an older MacOS version to maintain compatibility (GHA only provides 10.15 right now, where other services provide several options).

One finer point regarding this. In a Mac wheel, the 10_15 is actually the version of the SDK used to compile Python, which can differ from the OS X version itself. Last time I checked, for GitHub actions, the SDK was actually 10_13 on a 10_15 OS X.

But that's kind of a moot point because 10.13 is still such as recent version, all things considered. Most Mac OS wheels that I see have a 10.9 or even 10.6 sdk tag in them (example: https://pypi.org/project/cryptography/#files). So, your point is a valid one.

[nitzmahone]

That's good to know- I haven't played with publishing non-universal Mac wheels at all, so I'd been wondering how that worked there...

[matthewfeickert]

Apologies for just dropping in, as I see that @bsolomon1124 has already done some very nice work on PR #407, but you might be interested in looking at cibuildwheel for which there is a GitHub Action Workflow walkthrough thanks to @henryiii. This has been used quite successfully in Scikit-HEP to build lots of wheels for projects. For example, boost-histogram has wheels for Python 2.7 through Python 3.8 for Linux, macOS, and Windows.

[nitzmahone]

addressed in #407 in 5.4b1