Cannot add setup.py through the PEP 517 hook

[agronholm]

Using flit as the build back-end, there seems to be no way to specify that I want to generate a setup.py. For wheel, this is important to have for now. The help for flit build also incorrectly states that generating a setup.py is the default. Would you consider adding a configuration option in pyproject.toml for this, under [tool.flit.sdist]?

[gaborbernat]

Why would you need setup.py? I think the documentation is just outdated after implementing PPE-660.

[agronholm]

This is why.

[gaborbernat]

I don't follow, can you please summarize and state clearly. There is a lot of parallel discussion going on there.

[agronholm]

There is at least one downstream integrator who cannot yet handle the PEP 517 process in their automation, necessitating a setup.py in the sdist. I am trying to get more information to determine how pressing this need is.

I can work around this by using flit build instead of the standard build tool, so it's not a major issue.

[gaborbernat]

I'm personally would vote for flit build to also drop generating setup.py as shouldn't be needed in any modern build system. I personally would say flit is not intended to be used through old legacy build systems, and if you must pin it to an old version that still generates that. 🤔 I'll let other maintainers chime in though.

PS. I don't think we intend to support legacy build systems.

[agronholm]

as shouldn't be needed in any modern build system

I know – the comment I linked to was referring to a legacy build system.

[agronholm]

PS. I don't think we intend to support legacy build systems.

Isn't support for legacy installers exactly why the --setup-py option was added to flit?

[agronholm]

It seems I cannot use flit build, as I am using flit-scm as the actual backend so flit build is left missing the generated _version.py file.

[jameshilliard]

I don't think we intend to support legacy build systems.

IMO providing more/better migration paths is quite important to get adoption of newer build systems/tools, being overly eager to drop legacy support is a mistake IMO, similar to how not providing backwards compatibility was a huge problem with how the python2->python3 upgrade itself was rolled out. Many packages were only able to migrate once improved support/tooling for single source python2/python3 code-bases was introduced as it allowed for incremental migration(often the only feasible way to migrate larger projects).

In the same way build tools especially should try to simultaneously support legacy/modern installation methods for the purpose of simplifying migration(by allowing it to be done more incrementally), being overly eager to drop backwards compatibility ultimately slows down the ecosystem upgrade process IMO.

[gaborbernat]

flit has supported generating setup.py for at least five years, and PEP-517/518 has been around for almost that long. https://www.python.org/dev/peps/pep-0517/ was created in September 2015, flit was published in March 2015. We're in 2022. So I'm not sure how wanting to drop support for something for what there's been a new standard for five plus years now should be considered as eager. 🤔 What would be a reasonable timeline here, another five years?

[agronholm]

For my immediate needs, I think I need to drop flit-scm and go back to writing versions directly to src/wheel/__init__.py in addition to tagging them on git.

[jameshilliard]

flit has supported generating setup.py for at least five years, and PEP-517/518 has been around for almost that long. https://www.python.org/dev/peps/pep-0517/ was created in September 2015, flit was published in March 2015. We're in 2022.

I mean, PEP517 tooling is really only just starting to mature somewhat, even basic core functionality like installer cli support is still a work in progress for example.

What would be a reasonable timeline here, another five years?

It looks like python 3.11(last version compatible with distutils) isn't going to be EOL until 2027, so 5 years sounds about right for the upper range it would make sense to keep legacy install support around for.

We're still in the process of purging the last remnants of python2 from buildroot ourselves, legacy build tooling tends to stick around for a while so it's helpful to support it as long as feasible for incremental migrations.

[gaborbernat]

even basic core functionality like installer cli support is still a work in progress for example.

I'm pretty sure you can use pip to install wheels, this is what virtualenv has been doing since day 1 - https://github.com/pypa/virtualenv/blob/main/src/virtualenv/seed/embed/pip_invoke.py#L36-L38. I'd consider pip very mature. You could also use pip to build wheels for a similarly long timeframe; with build being around for a year and a half too. That being said I'm not saying flit<3.6.0 should not support the old ways, and you're free to use those old versions; but I'm not sure why we need to keep around an old legacy build system for 12 years. I don't see how is relevant the deprecation of the distutils with this topic.

[jameshilliard]

you can use pip

Last I checked pip cross compilation support is very broken/non-existant in general.

I'd consider pip very mature.

It tends to handle any non-standard install scenarios where all aspects of an installation need to be tightly controlled(like cross builds/installs) poorly in my experience.

I'm not sure why we need to keep around an old legacy build system for 12 years

To make things easier for those on older non-EOL python versions, at least until tooling/migration paths mature a bit more would be a reason, it's somewhat subjective though as different users have different needs.

[gaborbernat]

Last I checked pip cross compilation support is very broken/non-existant in general.

Couldn't you use pip under the same python you're building the wheel for?

It tends to handle any non-standard install scenarios where all aspects of an installation need to be tightly controlled(like cross builds/installs) poorly in my experience.

It's the de facto standard for like 20 years, not sure how much would the installer project help 🤔 here.

To make things easier for those on older non-EOL python versions, at least until tooling/migration paths mature a bit more would be a reason, it's somewhat subjective though as different users have different needs.

PEP-517/518 works independently of the python version. It's not tied to a language feature, so why would the build system be tied to an interpreter version.

[jameshilliard]

Couldn't you use pip under the same python you're building the wheel for?

Nope, when cross compiling you have to build using the host python, you can't run the target python on the host(since target python may be built for an entirely incompatible architecture).

It's the de facto standard for like 20 years, not sure how much would the installer project help 🤔 here.

For normal python application development sure, but I don't think pip ever really worked right for cross builds+installs. Installer seems to be the closest to what's needed for cross compilation installs.

PEP-517/518 works independently of the python version. It's not tied to a language feature, so why would the build system be tied to an interpreter version.

Legacy distutils deprecation is tied to the interpreter version from my understanding.

[gaborbernat]

Legacy distutils deprecation is tied to the interpreter version from my understanding.

Yes, but the build logic one should use is not tied to distutils. PEP-517+518 has been the recommended way of building packages for over six years now, and those PEPs at no point use or mandate any dependency on distutils, so IMHO this point is orthogonal to this topic.

Nope, when cross compiling you have to build using the host python, you can't run the target python on the host(since target python may be built for an entirely incompatible architecture).

Sure, if you don't want to setup the same python on the build machine as the target interpreter. But IMHO that's a bad idea, you should always build/install on the build machine with the same python interpreter as the target machine because otherwise, your wheel build might not generate a compatible wheel.

[jameshilliard]

Yes, but the build logic one should use is not tied to distutils.

Sure, I just mean that would give a potential EOL timeline for distutils based build system fallbacks.

PEP-517+518 has been the recommended way of building packages for over six years now, and those PEPs at no point use or mandate any dependency on distutils, so IMHO this point is orthogonal to this topic.

I mean the setup.py generator makes a distutils fallback for legacy setups, not the new PEP-517+518 setup of course. The removal of distutils fallbacks was only done very recently and before much of the tooling needed for non-standard PEP517 build/install setups matured(the PEP517 toolchain requirement is very new effectively since there was a fallback before).

Sure, if you don't want to setup the same python on the build machine as the target interpreter.

So of course we have to cross compile the target python interpreter on the build machine as well...the problem is you can't actually run the target python interpreter on the build(host in buildroot terminology) machine so we also have to build a host python interpreter(from the same upstream source but targeting the architecture of the build/host machine) for the build/host system architecture, this host interpreter is basically a part of the host cross-toolchain.

But IMHO that's a bad idea, you should always build/install on the build machine with the same python interpreter as the target machine because otherwise, your wheel build might not generate a compatible wheel.

Except that's not really possible, say I'm building for a aarch64 target from a x86_64 build host, in order to actually run for example a setuptools build+install setup.py or PEP517 build+install I need an interpreter that can actually run on the x86_64 build/host system but generates artifacts that run on aarch64 target system.

Doing this is a bit tricky for a number of reasons since we have to do things like override the sysconfig and some other stuff so that any c extensions get built against the aarch64 target interpreter/libraries instead of the x86_64 build/host interpreter(the host toolchain that actually runs the build scripts). We also have to override the installation location so that it installs into the target rootfs instead of the host toolchain site-packages. From my understanding these sort of overrides required for cross-compilation aren't really supported by pip.

[pradyunsg]

You can pass --setup-py to flit build, telling it to generate an sdist with the file.

The help for flit build also incorrectly states that generating a setup.py is the default.

Yup! We missed a line!

flit/flit/__init__.py

Line 85 in 68ed852

"This is the default for now, but will change in a future version."

Would you consider adding a configuration option in pyproject.toml for this, under [tool.flit.sdist]?

One implementation-related concern here is that flit_core does not know how to generate the setup.py file; that code lives in the command line only + Python 3.7+ flit package.

Is there a reason that you can't run flit build --setup-py? I don't think we want to encourage folks to be generating these compatibility shims, and the relevant functionality is available to package authors already, if they really need it.

[agronholm]

Is there a reason that you can't run flit build --setup-py? I don't think we want to encourage folks to be generating these compatibility shims, and the relevant functionality is available to package authors already, if they really need it.

Yes, as I stated before: I am using flit-scm as the actual backend to avoid having to update the version number in two places. This is my preference, but I can live without this convenience if this feature request is not accepted.

[pradyunsg]

I am using flit-scm

Ah, I missed this while skimming through the discussion.

Like Bernat, I lean toward not adding the setup.py generation logic to more places. Unlike Bernat, I think we should drop support from generating the setup.py stub sometime late this year/early next year (i.e. not immediately) since we just flipped the default to not generating the setup.py file by default.

[agronholm]

Yes, but the build logic one should use is not tied to distutils.

Sure, I just mean that would give a potential EOL timeline for distutils based build system fallbacks.

PEP-517+518 has been the recommended way of building packages for over six years now, and those PEPs at no point use or mandate any dependency on distutils, so IMHO this point is orthogonal to this topic.

I mean the setup.py generator makes a distutils fallback for legacy setups, not the new PEP-517+518 setup of course. The removal of distutils fallbacks was only done very recently and before much of the tooling needed for non-standard PEP517 build/install setups matured(the PEP517 toolchain requirement is very new effectively since there was a fallback before).

Sure, if you don't want to setup the same python on the build machine as the target interpreter.

So of course we have to cross compile the target python interpreter on the build machine as well...the problem is you can't actually run the target python interpreter on the build(host in buildroot terminology) machine so we also have to build a host python interpreter(from the same upstream source but targeting the architecture of the build/host machine) for the build/host system architecture, this host interpreter is basically a part of the host cross-toolchain.

But IMHO that's a bad idea, you should always build/install on the build machine with the same python interpreter as the target machine because otherwise, your wheel build might not generate a compatible wheel.

Except that's not really possible, say I'm building for a aarch64 target from a x86_64 build host, in order to actually run for example a setuptools build+install setup.py or PEP517 build+install I need an interpreter that can actually run on the x86_64 build/host system but generates artifacts that run on aarch64 target system.

Doing this is a bit tricky for a number of reasons since we have to do things like override the sysconfig and some other stuff so that any c extensions get built against the aarch64 target interpreter/libraries instead of the x86_64 build/host interpreter(the host toolchain that actually runs the build scripts). We also have to override the installation location so that it installs into the target rootfs instead of the host toolchain site-packages. From my understanding these sort of overrides required for cross-compilation aren't really supported by pip.

Where could I get details about your process for this?

[takluyver]

Whew. There's quite a lot written here before I read any of it, so apologies if I've missed some of it.

It seems that the motivation for this request is coming from @jameshilliard's issues with PEP 517 backends and tooling. We're working on this - in particular, flit_core now vendors tomli to avoid the circular dependency. I'd rather focus on moving forwards and fixing the issues with the new way of doing things rather than making it easier to go back to the old way of doing things. We've just turned setup.py generation off by default, it's never been part of flit_core, and I would hope to get rid of it entirely within another year or two. Every time someone complains about its limitations, I get an itch to rip it out faster.

If you are considering using Flit to package wheel as a way out of the dependency cycle with setuptools - or for any other piece of packaging infrastructure that might be involved in bootstrapping - I would really recommend not involving flit-scm, on the KISS principle. I can understand the desire, but I think packaging tools themselves should avoid extra complexity in their packaging. I wasn't aware of flit-scm until just now, and I'm mildly annoyed that it's using the name of Flit like that.

[jameshilliard]

One implementation-related concern here is that flit_core does not know how to generate the setup.py file; that code lives in the command line only + Python 3.7+ flit package.

Yeah, having it in flit_core I think would be more useful than having it in flit, then I think flit_core could be used to generate the setup.py file for building+installing itself I think(at least until there is a better way to do this).

Is there a reason that you can't run flit build --setup-py? I don't think we want to encourage folks to be generating these compatibility shims, and the relevant functionality is available to package authors already, if they really need it.

This functionality is most useful for downstream integrators rather than package authors IMO. Since the need for a setup.py shim is determined less by upstream development requirements but more by downstream build+install integration requirements.

[jameshilliard]

Where could I get details about your process for this?

The infrastructure code for this is pretty all in pkg-python.mk.

[SpecLad]

FWIW, I believe that Buildroot is now able to build flit-based projects without the use of setup.py (as of commit buildroot/buildroot@8b4f831), so the original reason for this request is no longer relevant.

[jameshilliard]

I believe that Buildroot is now able to build flit-based projects without the use of setup.py (as of commit buildroot/buildroot@8b4f831),

Well basic flit support for target packages was added prior to that commit, but that one reworked things further to use new bootstrapping method for our host toolchain.

so the original reason for this request is no longer relevant.

Well it may still be desirable for packaging things for distros that haven't yet introduced proper flit infrastructure support which can be somewhat complex depending on requirements.

[takluyver]

I'm going to close this now. I think we have a better bootstrapping story than when we were having the discussion, and I assume that in the last two years downstreams have worked out better how to build packages without setup.py. I'll reopen it if anyone strongly disagrees, but I was against expanding generating setup.py then, and I'll be against it now. 😉

One final remark on re-reading this thread. There was some disagreement on whether we were being too hasty to break the old way of doing things. And perhaps we were! But I also suspect that downstream packaging ecosystems only really start adapting when the old way breaks, so leaving compatibility machinery in for longer doesn't actually help that much, unless we're willing to keep it and support it for ever.

I don't think downstreams are being lazy - you have a lot of packages to deal with, you want them to be as similar as possible and to work from one release to the next with as little manual intervention as possible. It all makes sense! But perhaps when situations like this occur, we can understand why it seems to downstreams like a very quick change, and downstreams can understand why it doesn't seem that way to us upstream. 🙂