Don't generate setup.py by default.

[takluyver]

Autogenerating setup.py was a workaround to let pip install from source before PEP 517. pip supports PEP 517 since 19.0 (January 2019). Of course it supports installing wheels since much earlier (2013), and this is the default installation method. So I'm gradually moving towards getting rid of the generated setup.py file.

In Flit 3.4 (2021-10-10), I added a message saying that the default would change, and adding a (no-op) --setup-py option. This PR will make --no-setup-py the default behaviour, and update the message accordingly.

I haven't decided when to merge this yet. The flit command line tool is meant as a developer tool, so I think it's reasonable to do changes like this relatively quickly (unlike in the lower-level flit_core package). But the purpose of adding the message in 3.4 is to give people a chance to be aware of the change coming and react to it.

[jameshilliard]

This seems to be causing problems as there still aren't alternatives to python setup.py build AFAIK for PEP 517(maybe I missed how one is supposed to do a regular build here).

I was trying to package a python package without a setup.py for buildroot which uses separate build and install stages but I haven't manged to figure out how to do a build without an install without the generated setup.py.

[takluyver]

The generic answer is the build package, which should be able to invoke the build backend for any Python project, based on the information specified in its pyproject.toml (falling back to setuptools & setup.py if nothing else is specified).

Python packaging is also moving towards separate build & install stages, with the wheel package format as the boundary between them, so build will give you a wheel. The installer package is intended to be the low-level counterpart to build, which takes a wheel and installs it, but it doesn't have a CLI yet (there's a PR at https://github.com/pradyunsg/installer/pull/66). You can write a Python script to call it, or use pip for now to install from a wheel.

[jameshilliard]

The generic answer is the build package, which should be able to invoke the build backend for any Python project, based on the information specified in its pyproject.toml (falling back to setuptools & setup.py if nothing else is specified).

So to clarify we're not trying to build a package like a sdist or a wheel, we pull in the sdist and are trying to do a basic build in place like python setup.py build does.

Python packaging is also moving towards separate build & install stages

Hmm, I wonder why it's unclear how to do this when it used to be straight forward with distutils/setuptools. Wheel is used for distributing pre-built packages on sites like pypi which is not what we're trying to do. We're just trying to do the compile/build stage from the existing sdist which comes before the install, this would be the stage say where any c extensions would get built but before the package is installed into the target site-packages. This works fine with setuptools and distutils at least but it's unclear how to do this with PEP-517.

The installer package is intended to be the low-level counterpart to build, which takes a wheel and installs it, but it doesn't have a CLI yet (there's a PR at pradyunsg/installer#66).

Well we're not trying to use wheels at all so I'm not sure that would be right, we're basically trying to do a standard in tree build of the sdist in preparation for install, we're only trying to do a build, we're not trying to create artifacts for distribution outside of buildroot.

You can write a Python script to call it, or use pip for now to install from a wheel.

Hmm, it's unclear what we should actually be calling, and pip doesn't seem to provide a build(without packaging) option either from what I can tell, we've always had to directly invoke it from distutils/setuptools in the past. We don't even support pip for builds due to it not being cross compilation friendly last I checked.

[takluyver]

Wheel is used for distributing pre-built packages on sites like pypi which is not what we're trying to do. We're just trying to do the compile/build stage from the existing sdist which comes before the install, this would be the stage say where any c extensions would get built but before the package is installed into the target site-packages.

The scheme Python packaging has been working towards for the last few years is to use the wheel as the standard interface between the build step and the install step. So whatever build system the project uses - such as Flit - produces a wheel with the interface in PEP 517, and that defines what is to be installed.

I.e. the wheel is the concrete representation of the stage you describe after building C extensions etc. but before the package is installed. The wheel represents all the files that should be installed, without the build tool having to know anything about the install location(s).

[jameshilliard]

So whatever build system the project uses - such as Flit - produces a wheel with the interface in PEP 517, and that defines what is to be installed.

So I guess it's expected that distros round trip the stdists through wheels for all package builds I guess? I did come across this which seems similar to how we normally do builds.

The wheel represents all the files that should be installed, without the build tool having to know anything about the install location(s).

Hmm, I thought using wheels caused issues with cross compilation unless they are pure python(and even then I think pip didn't play nice in general with cross compilation), although things may have changed. I guess I probably need to wait for the installer cli interface though, any idea if installer has been tested in cross compilation scenarios?

Historically we've had to hack up the internals to get python's build systems to place nice with cross compilation in general as cross compilation seems to not get a lot of testing in that area from most python packaging related projects.

[takluyver]

So I guess it's expected that distros round trip the stdists through wheels for all package builds I guess?

That's the idea, yes. 🙂

I did come across [build_editable] which seems similar to how we normally do builds.

I think that's a red herring. 'Editable' installs are meant for development: instead of installing the package files, you install a little shim which adds the source location to the import path, so you can change the source and use it without having to reinstall it. As part of that, it can do an in-place build if there's anything that needs building.

It's possible you could figure out some way to use that for downstream packaging, but it's quite different from the intended use case, so you'd be swimming against the stream. The build_editable hook is also optional for backends, whereas build_wheel in PEP 517 is required.

I thought using wheels caused issues with cross compilation unless they are pure python(and even then I think pip didn't play nice in general with cross compilation),

Pure Python wheels, where the filename ends like -none-any.whl (no ABI requirement, any platform) should be fine - you can build them on one platform and install on another. For packages with compiled parts... I don't really know, to be honest. You're quite right, this is not something Python packaging is adapted for. Our normal expectation is that if something needs to be specifically built for a given platform, you build it on that platform. https://www.piwheels.org/ builds many thousands of packages on Raspberry Pis, for instance.

any idea if installer has been tested in cross compilation scenarios?

The install is the easy bit - the wheels already contain the compiled libraries for a specific platform, and installing is just a matter of unpacking them to the right locations (plus generating some script wrappers and fiddling with a bit of metadata). Any complexity involved in cross-compilation is on the build side, to get the wheel.

[jameshilliard]

That's the idea, yes. 🙂

Seems a bit odd to me but I suppose it's possible it could work, the how part is just not all that obvious for cross compilation scenarios like this.

Pure Python wheels, where the filename ends like -none-any.whl (no ABI requirement, any platform) should be fine - you can build them on one platform and install on another.

Yeah, I don't think those are that big an issue. We just don't use them currently at all since we prefer using sdists.

For packages with compiled parts... I don't really know, to be honest. You're quite right, this is not something Python packaging is adapted for. Our normal expectation is that if something needs to be specifically built for a given platform, you build it on that platform. https://www.piwheels.org/ builds many thousands of packages on Raspberry Pis, for instance.

Yeah, which is an obviously wrong assumption, but it has unfortunately managed to get baked into a lot of python's packaging/build tooling.

The install is the easy bit - the wheels already contain the compiled libraries for a specific platform, and installing is just a matter of unpacking them to the right locations (plus generating some script wrappers and fiddling with a bit of metadata). Any complexity involved in cross-compilation is on the build side, to get the wheel.

Well install is often much less simple with cross-compilation than one would often assume since the install location(site-packages for the target python interpreter) also differs from that used by the running python interpreter(the host python interpreter). There's a good bit of complexity on both sides here I think.

By the way would it be possible to provide setup.py's for bootstrapping flit and tomli? Right now it's kinda a PITA to install them due to the recursive dependency issue.

[takluyver]

Yeah, which [build on the appropriate platform] is an obviously wrong assumption, but it has unfortunately managed to get baked into a lot of python's packaging/build tooling.

TBF, from a Python packaging point of view, it's not obviously wrong. I'd estimate that ~90-95% of packages on PyPI are pure Python, where we don't have to worry about cross compilation at all. And for the packages that do need compilation, 'build it on the relevant platform' generally works well enough for ~95% of what the people we hear from are interested in.

If you're interested in discussing how Python packaging can improve on this, then the Packaging category on the Python Discourse forum is probably the place to do so.

Well install is often much less simple with cross-compilation than one would often assume since the install location(site-packages for the target python interpreter)

Sure, but so long as you know what the right target directories are, it's just a matter of specifying them in the right place. installer can deal with this - it takes a 'scheme' dictionary of directories to install to.

By the way would it be possible to provide setup.py's for bootstrapping flit and tomli? Right now it's kinda a PITA to install them due to the recursive dependency issue.

I'm sorry about this difficulty, and I've done my best to document how to deal with it:

https://flit.readthedocs.io/en/latest/bootstrap.html

I hope that one day, there will be a TOML parser in the Python standard library. Until then, it's hard to avoid needing a bit of a manual workaround. There's more discussion about this in #462.

I'm afraid I'm not prepared to reintroduce setup.py - we've done a lot of work in the last few years to build packaging infrastructure on a solid base of specifications, rather than 'whatever setuptools does'. The author of tomli may choose to, but so far (in #462), he seems to be on board with the idea of bootstrapping without setuptools. Of course you can patch in setup.py to either project if that's what you need to build them in your setup.

[jameshilliard]

I'm sorry about this difficulty, and I've done my best to document how to deal with it:

Reading that it says flit is itself packed with flit, so wouldn't that mean you just need to set --setup-py during the flit and tomli build process? Obviously this would need to be done for the sdist generation as we wouldn't be able to do it downstream due to the chicken and egg problem(we also don't support pip so we can't just use a prebuilt whl either).

Until then, it's hard to avoid needing a bit of a manual workaround.

I don't see how it's hard to avoid, seems setting --setup-py is all that's needed for flit and tomli sdist builds.

You could also just copy flit_core from the source directory

This is the only thing that seemed to work without a setup.py(since we don't have whl infrastructure bootstrapped already), but it's very brittle and not ideal for bootstrapping.

[takluyver]

There are several reasons that I don't want to do that.

The most obvious & least important is that setup.py generation is in the higher-level flit package, which shouldn't be involved in building flit_core at all. That's unimportant, because the relevant code could be moved to flit_core, but it's not just a matter of turning the switch on.

Next, for the dependency cycle between flit_core & tomli, having a setup.py with flit_core doesn't actually solve it. flit_core can already build itself without tomli - I've taken care to ensure this is possible. But it has a runtime dependency on tomli, so you need tomli to build tomli. Having a setup.py with tomli would be a workaround for that (but not the only one).

Finally, I want to have Flit & the ecosystem of newer tools & standards we've developed standing firmly on their own feet, not propped up by setuptools right at the base. The --setup-py option, and the code in flit to support it, is a temporary measure which will eventually go away. It's already gone from always-on to opt-out to opt-in, and that trajectory is going to continue. I don't like the signal that releasing flit_core with a setup.py would send. And if I add it temporarily, people will rely on it, and this discussion will happen all over again when I remove it.

Again, though, if you want to add a setup.py as a patch in your own packaging, that's fine. I know that some workaround is needed for bootstrapping, and while that's not the one I'd pick, I won't fault you if you want to. It can probably be pretty simple and need only minimal maintenance. But whatever decisions & maintenance it does require is on you (your project, not you personally).

Some other resources you might find interesting:

• https://github.com/FFY00/python-bootstrap (I just saw this recently)
• A talk on how Fedora is dealing with the new world of PEP 517: https://www.youtube.com/watch?v=pJlqB-PA96E

This is the only thing that seemed to work without a setup.py(since we don't have whl infrastructure bootstrapped already)

For simple packages like flit_core & tomli (pure Python, no scripts), installing the whl is just a matter of unzipping it to the site-packages directory. This is how I'd approach bootstrapping.

[jameshilliard]

But it has a runtime dependency on tomli, so you need tomli to build tomli. Having a setup.py with tomli would be a workaround for that (but not the only one).

I was working on fixing that as well....why not bundle tomli with flit like pip does?

Finally, I want to have Flit & the ecosystem of newer tools & standards we've developed standing firmly on their own feet, not propped up by setuptools right at the base.

So make migration/bootstrapping as difficult as possible? That seems rather counterproductive....

I don't like the signal that releasing flit_core with a setup.py would send. And if I add it temporarily, people will rely on it, and this discussion will happen all over again when I remove it.

I don't get this...you want people to migrate but you want it to also be as painful as possible by making even a simple installation super difficult for anyone that has custom distro tooling(by creating recursive dependency issues)?

I did manage to hack something together temporarily but it's really not ideal.

[takluyver]

I was working on fixing that as well....why not bundle tomli with flit like pip does?

My experience with Linux distros is that the people who care most about this kind of bootstrapping also insist on unbundling dependencies, so bundling makes more work for downstreams, not less. I might revisit that if there are more downstream ecosystems that are fine with bundling... but if I did that, I'd expect pushback from the ecosystems that I know dislike bundling. 🤷

So make migration/bootstrapping as difficult as possible?

I appreciate that not having a setup.py file makes your life more difficult, but I don't think this is 'as difficult as possible'. I've thought about bootstrapping, written docs, and offered suggestions here. Once more, this is how I would suggest downstreams deal with the tomli <-> flit_core dependency cycle:

• Package flit_core
  • Use its build_dists.py script to get a wheel. This doesn't require anything except Python. Then use unzip or pip to unpack that to site_packages. I could even add a bootstrap install script that can do this if you like.
  • Alternatively, copy the package directory to site-packages manually (but be aware pip won't know it's installed without the metadata from the wheel).
  • Alternative 2, add a setup.py yourself and run that.
• Package tomli
  • Install the flit_core package from the previous step
  • Make tomli importable from source (e.g. by setting PYTHONPATH)
  • Call the Python function flit_core.buildapi.build_wheel(some_tmp_dir) to make a wheel
  • Use unzip or pip to install the wheel.

I feel like your responses are gradually edging towards irritated grumbling that I won't do what you want, rather than having a productive discussion. I won't lock the thread yet, but please, if you're frustrated about this, go and do something else for a few hours rather than writing a fast, grumpy reply.

[jameshilliard]

My experience with Linux distros is that the people who care most about this kind of bootstrapping also insist on unbundling dependencies, so bundling makes more work for downstreams, not less. I might revisit that if there are more downstream ecosystems that are fine with bundling... but if I did that, I'd expect pushback from the ecosystems that I know dislike bundling. 🤷

I'd say while this is true for applications it's more often the opposite for build systems(good ones at least), the popular cross-language python based build system meson(used for building apps like systemd on all modern distros) has a very strict rule when it comes to dependencies which I agree with a lot(for python based build tools at least):

There have been many Python programs that are difficult to maintain on multiple platforms. The reasons come mostly from dependencies. The program may use dependencies that are hard to compile on certain platforms, are outdated, conflict with other dependencies, not available on a given Python version and so on.

Meson avoids dependency problems with one simple rule: Meson is not allowed to have any dependencies outside the Python basic library. The only thing you need is Python 3 (and possibly Ninja).

I appreciate that not having a setup.py file makes your life more difficult, but I don't think this is 'as difficult as possible'. I've thought about bootstrapping, written docs, and offered suggestions here. Once more, this is how I would suggest downstreams deal with the tomli <-> flit_core dependency cycle:

Yeah, thanks for the suggestions, I've actually been experimenting a bunch, I'm mostly trying to see if I can find a more maintainable solution for buildroot than that pile of ugly hacks I came up with, ideally something that has a more incremental migration path.

Package flit_core
Use its build_dists.py script to get a wheel. This doesn't require anything except Python. Then use unzip or pip to unpack that to site_packages. I could even add a bootstrap install script that can do this if you like.
Alternatively, copy the package directory to site-packages manually (but be aware pip won't know it's installed without the metadata from the wheel).
Alternative 2, add a setup.py yourself and run that.
Package tomli
Install the flit_core package from the previous step
Make tomli importable from source (e.g. by setting PYTHONPATH)
Call the Python function flit_core.buildapi.build_wheel(some_tmp_dir) to make a wheel
Use unzip or pip to install the wheel.

I've made a few attempts at something along these lines but keep getting bogged down in dependency hell fairly quickly.

I feel like your responses are gradually edging towards irritated grumbling that I won't do what you want, rather than having a productive discussion. I won't lock the thread yet, but please, if you're frustrated about this, go and do something else for a few hours rather than writing a fast, grumpy reply.

Sorry, I'm really just trying to bounce ideas around to see what I'm missing and what might be missing on on the pep-517 build system side of things.

I'm usually pretty good with build systems in general(I help maintain a lot of packages in buildroot) but none of the approaches I've tried so far appeared to be all that maintainable when it comes to the integration with buildroot(which to be fair is a bit weird when it comes to python stuff).

It's not out of the norm for features we need for cross compilation, or features needed due to our distro tooling being a bit weird to be missing entirely from newer build systems(and it's not out of the norm to have features we need missing on older widely used build systems like meson either), sometimes it just takes a good bit of back and forth to find a good maintainable solution(which is what I'm trying to do).