Switch to setup.cfg for metadata specification?

[pfmoore]

See pypa/sampleproject#56 - should we be recommending that new projects put their metadata in setup.cfg rather than in setup.py?

[theacodes]

I can't think of a good reason why we shouldn't.

[dstufft]

Main reason against it would be that it's a relatively new feature and doesn't fail gracefully if someone has an older setuptools.

[theacodes]

Our guides generally recommend people install the latest versions, or did you mean for people installing the package from sdist?

[dstufft]

Yea, it affects both the author and random people installing the package from sdist. This also affects services like RTD (which just came up in #pypa yesterday).

[theacodes]

Hrm. that makes things a bit more tricky.

[pfmoore]

Good point. I guess it depends on the role of the PUG, to an extent. If we're intending to be a discussion of the options available, I strongly believe this should be mentioned (as it's far nicer to use than the setup() approach), although we'd probably mention both approaches in that case. If we're intending to be an opinionated guide, then I'd prefer that we emphasised "modern" practice, rather than older conventions, although we should note the compatibility issues. If we're looking to be a guide to how to best navigate all of the nasty pitfalls that people might hit when packaging Python projects, then we need to mention the new form, explain why it can cause problems, and recommend the older form "until a sufficiently large proportion of users have caught up".

Looks like setuptools has had this feature since Dec 2016. If we don't mention it now, when should we start to do so? (If anyone says "when it's the default setuptools in RHEL" I'll scream 😄) If there was a clear timescale (once RTD switch, or once pip 18 comes out with PEP 518 support for requiring a specific version of setuptools, or something) then I'd be OK with waiting.

One of the things that prompted me to flag this up is that as far as I know, this feature was introduced with very little publicity - which I'm frustrated about because it's such an improvement in usability (IMO). Functionally it makes little difference, but by emphasising declarative metadata it's a major shift.

[theacodes]

If we're intending to be a discussion of the options available, I strongly believe this should be mentioned (as it's far nicer to use than the setup() approach), although we'd probably mention both approaches in that case.

Yes, I think this is very appropriate for something that goes into the guides section.

If we're intending to be an opinionated guide, then I'd prefer that we emphasised "modern" practice, rather than older conventions, although we should note the compatibility issues.

The "tutorials" are where we get opinionated. I plan to use flit for the new tutorial for packaging items, so this particular discussion is non-applicable. I do think that the guides on setuptools can and should say "do it this way if you can, but you can also do this way if you must".

If we're looking to be a guide to how to best navigate all of the nasty pitfalls that people might hit when packaging Python projects, then we need to mention the new form, explain why it can cause problems, and recommend the older form "until a sufficiently large proportion of users have caught up".

This is good stuff for a discussion article, but it can also be accommodated in a guide on setuptools.

Looks like setuptools has had this feature since Dec 2016. If we don't mention it now, when should we start to do so? (If anyone says "when it's the default setuptools in RHEL" I'll scream 😄) If there was a clear timescale (once RTD switch, or once pip 18 comes out with PEP 518 support for requiring a specific version of setuptools, or something) then I'd be OK with waiting.

I look to the rest of the PyPA to figure this out. :)

[pfmoore]

I plan to use flit for the new tutorial for packaging items

I don't want to divert the subject here, but given that we're talking about when it's reasonable to recommend a new feature, is it really consistent to be hesitant about recommending a setuptools feature that's been out for 9 months which can't be used when installing from source with an older setuptools, and yet be willing to recommend flit as the build tool when you can't install from source with that unless you use the as-yet unreleased pip 10 with PEP 517 support?

[dstufft]

I don't think we should recommend flit until pip has PEP 517 support, and even then I'm not sure what level of uptake makes sense.

There are some differences of course, flit restricts itself to stuff that can ship universal wheels, so for most people pip install is going to get a wheel and if you're installing from a sdist and your pip is too old, you get a hard failure instead of with setuptools feature you just get an UNKNOWN project with version UNKNOWN and no files.

[pfmoore]

There's also some questions we'd have to consider in terms of our overall position on project layout. Flit is taking the view that it supports a single "best practice" approach to project layout, so we'd need to be sure the guide is consistent with that - specifically flit doesn't currently support the layout of having code in a src subdirectory (currently under discussion in #320 and pypa/flit#115). I suspect (as @takluyver noted on the flit issue) that this a wider question than flit or the PUG can decide unilaterally.

[theacodes]

(summing @ncoghlan for his take as well)

As part of the new tutorials we wanted to experiment with using some newer, more opinionated tools as the first ones that users interact with. The idea is that while they aren't perfect for every use case, they are simple and present less friction. For example, we've gotten good feedback on the new installing packages tutorial that uses pipenv instead of pip & virtualenv. None of this is to say that we want to not document the other tools - for example, I won't replace the current installing packages tutorial with the pipenv one until I'm certain that the pip & virtualenv instructions have a new home under guides.

That said, flit still may or may not be as ready as pipenv is to be that entrypoint. Perhaps this feature of setuptools is a viable alternative for having a simple introduction. I'm personally open to writing both and seeing what everyone thinks - but if y'all feel strongly right now that flit isn't the right choice I'll focus on a simplified and opinionated tutorial using setuptools.

[ncoghlan]

(Tangent: for RHEL, my own guidance is already:

1. Use OpenShift, RHSCL, or softwarecollections.org to get a newer base Python & pip
2. Always runs pip install --upgrade pip setuptools wheel as the first command in a new virtual environment

/tangent)

As far as publishing guidance goes, I think we need to split that into two distinct categories:

1. Advice for newcomers to Python who just want to get their project onto PyPI, so they and their close peer group can download it from there. For this use case, we should be recommending tools that offer the simplest and most straightforward publisher experience, even if we have to put a caveat on the advice that says "Note: end users running older installation tools that also choose not to use your uploaded wheel files may not be able to install source packages published this way. If this is something that matters to you, please see this alternate guide that provides guidance on maximising your ability to reach users of older clients and platforms"

2. Advice specifically for professional software developers that have a target audience of "All the Pythonistas, with correct software, no matter what platform they're running".

I see this distinction as the packaging equivalent of getting newcomers to Python to start with Python 3.6, and then only later giving them the bad news about all the features they're going to have to stop using if they want to make their libraries support older Python versions.

On the topic of src directories, I think that falls victim to the same kind of distinction in the learning curve: it's a code correctness issue that should NOT be inflicted on beginner programmers, just like unit testing, type checkers, etc. Beginner code is tested by running it and asking yourself "Did that do what I wanted it to do?". Professionals need to back the hell off and stop trying to teach people to run marathons at a point where they're still trying to grasp the basics of walking.

[pfmoore]

Thanks @ncoghlan - your comments on publishing advice basically match my position, just explained better ;-)

I guess the remaining question for me is whether the "basic publishing" guide should recommend setuptools-with-static-metadata or flit. Longer term, I'm sure that flit will be a better choice, but right now I'm concerned that it might be a bit too bleeding edge for beginners. And setuptools-with-static-metadata gives pretty much the same basic experience (put your data into a config file) while running with our recommended toolset (pip, twine, OK, we don't have a standardised "build a sdist" tool, sorry) and other community standards (tox, for example).

Regarding src directories, I disagree somewhat. Mostly because I think that by definition we're targeting programmers who have reached a point where they are packaging up code for (some level of) distribution, and therefore they should at least be aware of the ideas of documentation and automated testing - even if the PUG isn't the right place to discuss those topics. But when you start on automated testing, "the test tool can't find my code" is a very common problem (there are lots of StackOverflow questions on this). Structuring code with a src directory doesn't fix the problem, but it does make it more obvious what's going on and makes it easier to set things up correctly. But this is a side issue, and not directly relevant to this thread, so we can thrash this question out elsewhere.

[ncoghlan]

Catching up on the full thread: I do think it makes sense to coordinate with the ReadTheDocs folks, and make sure that any advice we're giving will work in their default docs build environments (I'd missed that concern when writing my initial reply).

Regarding whether it makes sense to recommend setup.cfg over setup.py at some point, I'm not sure that's actually a good idea, since:

1. setup.py is still needed anyway to actually provide the backend build interface
2. Not all settings can go in the setup.cfg file (e.g. entry points still needs to be in setup.py)

So we'd probably end up making the initial guidance more complicated, since it would need to explain what can go in setup.cfg, and what still needs to go in setup.py, vs just using setup.py for everything.

The reason I think flit is a potentially interesting future default recommendation for new users isn't because it uses a static configuration file instead of an executable build script: it's because it's genuinely simpler to get started with than setuptools, as it's far less configurable (no extension modules, no selective manifests, etc).

[pfmoore]

OK, those arguments seem convincing to me. For my personal use, I'll probably switch to declarative setup.py and hold off on flit till the ecosystem around PEP 517/518 has matured a bit, but I agree that it's not something that should go into the main guides. (I'll leave it to others to judge when the correct time to switch to recommending flit will be).

[pfmoore]

One thought - when we do switch our recommendations to flit, we should probably also switch pypa/sampleproject to use flit, too...

[ncoghlan]

Aye, agreed. We'll probably still want to keep an sample setuptools project around at that point, but we'd enhance it to do something like include a cffi or cython extension module, to help illustrate what the extra complexity enables.

For now though, I'm going to close this issue as "No, we'll stick with the focus on setup.py for the time being".

[takluyver]

[flit is] genuinely simpler to get started with than setuptools, as it's far less configurable (no extension modules, no selective manifests, etc).

The downside of recommending that, of course, is that if you later want those things, you have to switch your packaging tools to use them - it's harder to start simple and add complexity later.

One thing I'm considering is allowing a configurable build step, something like the rust build script you can specify for cargo. But I don't know much about compilation, so input from other people is welcome, whether you think it's something flit should deal with or not. Here's the discussion: pypa/flit#119