Allow setup.cfg as an alternative configuration file

[pytoxbot]

• Bitbucket: https://bitbucket.org/hpk42/tox/issue/297
• Originally reported by: @bittner
• Originally created at: 2015-12-16T17:01:01.612

(This issue comes from a comment in issue #185.) I'd love to see the tox configuration living in setup.cfg too in general, as an alternative.

It would allow a more elegant Python package configuration for developers: You could put configuration of all your tools into a single file. The ini-style allows for separate sections. I've described some considerations in a comment on issue 13.

A few projects already look into setup.cfg. flake8, behave, py.test, among others. Pylint is a prominent exception (its maintainer considers this idea harmful; declined PR; discussion).

[pytoxbot]

Original comment by @bittner

Would you guys prefer including all other test tools via testenv:'s in tox.ini instead? So, instead of a setup.cfg we would use a tox.ini as done in the behave-django project, for doctests, flake8, behave, etc.?

[pytoxbot]

Original comment by @RonnyPfannschmidt

I'm fine with a single file, but setup.cfg very certainly is not that file due to legacy and breakage potential

[pytoxbot]

Original comment by @The-Compiler

FWIW, in pytest abusing setup.cfg causes various kinds of pain (examples: 1311, 567), and there is some consensus to deprecate and drop setup.cfg support.

This seems very much a technical reason and not just a personal preference to not go down that rabbit hole again.

[pytoxbot]

Original comment by @bittner

Under the light of "move to a single config file for packaging and testing" (Holger): Allowing setup.cfg to house tox configuration should make sense, and should not be seen as "personal preference" (and would match existing py.test behavior as additional benefit).

We must respect other arguments in this discussion, but as long as they are personal preference and clash with future benefits of python we should be able to agree on a common ground. (I know this is difficult, because the whole discussion is about personal preference.)

[pytoxbot]

Original comment by @RonnyPfannschmidt

this will drag on a massive legacy, and now that pypa.json will come there will be another source of truth and setup.cfg might become obsolete

[pytoxbot]

Original comment by @hpk42

I understand that seing a "tox.ini" and "pytest.ini" signals useful information. On the other hand i'd like to see python move to a single config file for packaging and testing because i dislike the proliferation of config files. If we today say we want to have separate files we'll have setup.cfg, setup.py, pytest.ini, tox.ini to just distribute a single-module and a single-test module. Isn't this a bit much overhead/cruft? Somehow having a, say, setup.cfg which incorporates all the information seems leaner. Given some practise it means you will check for a "tox" section in that file instead of looking for the filename. Or just issuing "tox -l" and see. This can be automated. And to use setup.cfg is just a practical decision because we can't change the usage of this file easily and i always prefer incremental solutions instead of "everyone has to change".

[pytoxbot]

Original comment by @warsaw

@The-Compiler A couple of minor details in a moment, but I think you've nailed the essential bits. I do the same thing - when I see a tox.ini in an upstream project, there's no guessing as to how to run the test suite, and that is a huge win for Python in general. I've long pined for a consistent interface to test discovery, and there's more to be done with the newer metadata formats (i.e. it would be nice to have a declarative way to know), but as tox becomes more prevalent, it's becoming less mysterious how to run the tests. I used to thing setup.py test would be that interface, but for lots of reasons, it won't and probably shouldn't be. Can you tell I'm a huge fan of tox? :)

A few details: our source package git branch does include the full upstream source, and during package build we also have the full source available, albeit with any Debian patches already applied (though with pytest 2.8.5 we have only one such patch left). However, we can't use tox for technical reasons in our build tools, something I plan on working to fix at some point. (Our recommended Python build helper supports tox, but it needs to plumb through additional argument passing, e.g. --sitepackages). So for now, what I generally do is pluck out the [testenv]commands bit and put that in our build rules file. That's what I did in the pytest case. I just missed that the [pytest] section of the tox.ini was relevant. You're spot on for that point.

Unrelated to the issue we discussed (and for you help... thanks!) was that we cannot run the tests from the top-level source tree. I was doing that originally and got three mysterious failures in test_genscripts. I was about to give up on those tests (as I understand it, genscripts is deprecated anyway), but someone else in Debian helped debug this issue and now we cd to the temp directory when running the tests and they now pass.

One last thing, I've noticed that the pytest suite doesn't clean up all its tempdirs. Normally, our builds run in overlays so can't mess with the host filesystem, but this isn't the case for /tmp. For this reason I mktemp a separate tempdir and export $TMPDIR to run the test suite. It still leaves cruft, but it's easier to clean up after the fact.

Thanks for all your help! We now have 2.8.5 in Debian.

[pytoxbot]

Original comment by @The-Compiler

Some other thoughts:

Yesterday I helped @warsaw debug an issue where a pytest test failed when packaging for Debian. If I understood the issue correctly, Debian doesn't use tox to run the tests when packaging, so they didn't have the tox.ini around, which made the tests fail because of a missing option. Principle of least surprise, IMHO - you wouldn't search pytest configuration in tox.ini (if you don't know), just like you wouldn't search tox configuration in setup.cfg.

Also from my experience contributing to, say, about 20 python projects (quite possibly more), I see setup.py test used very rarely, and tox used extensively. If I see a tox.ini, I immediately know "hey, I can run tox to test stuff without having to see how things work for this project". If the tox configuration is hidden in setup.cfg, again, I'll have to search for it in places I wouldn't expect it to be.

[pytoxbot]

Original comment by @RonnyPfannschmidt

i think its a considerable dirty and problematic solution to just drop random things into the namespace of a config file of a different tool,

i already find i fairly problematic that pytest/flake8 configuration inside tox.init has no prefix for the configuration key

a much more preferable solution would be a standardized folder that tools can choose adapt or a per tool convention on adding other tools to the configfile

[pytoxbot]

Original comment by @hpk42

ah. in any case i agree it's a good idea to have a single file configuring packaging and testing. Ronny why do you think it's a bad idea to have tox config in setup.cfg? I guess we should at some point make it more official, probably by starting to discuss on distutils-sig.

[pytoxbot]

Original comment by @bittner

@hpk42 No, not tox. Only py.test does it. See _pytest/config.py (line 1121+).

[pytoxbot]

Original comment by @hpk42

not sure i follow. Isn't tox currently looking into setup.cfg already? I think you can define [tox] and [testenv] sections just fine there but maybe i misremember.

[pytoxbot]

Original comment by @RonnyPfannschmidt

Still a dirty solution that just makes something easy at first glance instead of simple

[pytoxbot]

Original comment by @bittner

Maybe I should give some more background information, so people can understand my motivation (and the one of those that share my opinion):

When you look at a typical Web development project you have tons of configuration files, the majority of them from frontend technologies (in agency projects I have participated with). The project repository root is polluted by files, which is just "noise" that makes your project less maintainable:

bower.json
config.rb
.coveragerc
.csscomb.json
Dockerfile | Vagrantfile
.eslintrc
Gemfile
Gemfile.lock
gulpfile.js
lint.yml
package.json
pylintrc
scss.mustache
setup.cfg
spritesmith-retina-mixins.template.mustache
tox.ini

I believe we "backend people" should know better. In projects that I do at least .coveragerc, pylintrc and tox.ini are only symlinks to setup.cfg. Some tools (behave BDD testing, flake8 and pytest) read their configuration from setup.cfg already.

Yes, setup.cfg was intended as a configuration file for distutils/setuptools, and it still is. And the good news is tox also integrates nicely into setuptools.

This way you can run your tests with python setup.py test, the standard approach. And tests run the way you want your project to be tested (e.g. using tox). No need to train new developers, no need to write extensive test instructions. It's just obvious, because the project follows an established standard approach.

So, what if in a typical project you have only a single file that you look into to find out what the test setup is like? That could be appealing to a couple of folks out there, myself included. For excessive use, like in Florian's qutebrowser project, you can still have this split up, which makes perfectly sense when you have such a huge amount of configuration data.

[pytoxbot]

Original comment by @The-Compiler

Sure, see https://github.com/The-Compiler/qutebrowser

Please keep in mind that I'm suggesting an alternative to the current approach, not a replacement.

I see that, but I'm still questioning if it's worth having the additional complexity in all those tools ;)

[pytoxbot]

Original comment by @bittner

Can you share what you have in a 243 lines tox.ini file? And 40 lines of pytest.ini (which you can put in tox.ini, IIRC). Pylint configuration is borked, agreed, everybody knows that; that's why projects like Prospector pop up.

Here is a working sample configuration for a bunch of test tools. 27 lines total; note that I have symlinks for .coveragerc, pylintrc, tox.ini pointing to setup.cfg. What features do I miss out that require more configuration?

Please keep in mind that I'm suggesting an alternative to the current approach, not a replacement.

[pytoxbot]

Original comment by @The-Compiler

Something about our configurations is clearly different then ;)

   18 .coveragerc
   65 .pylintrc
   40 pytest.ini
  243 tox.ini
  366 total

[pytoxbot]

Original comment by @RonnyPfannschmidt

ftr. by now i also consider that py.test looks in more than pytest.ini a problem

[pytoxbot]

Original comment by @bittner

@The-Compiler The "all-code-in-one-file" allegory is not correct.

Configuration for testing in a single location, especially when several different tools are used on a single project, has a distict advantage: You can compare the settings of the different tools in use. You have all your test-related settings in a single place.

I don't think this file will ever have more than 20 lines (i.e. half a screen page), even with a couple of tools using.

[pytoxbot]

Original comment by @bittner

Doing this is a bit tricky, though. It requires giving up some elegance in config.py where tox.ini is defined as a default for the -c command line argument. Respecting "several defaults" requires some spaghetti code later in the process. Probably on line 218. Not very elegant. 😧

[pytoxbot]

Original comment by @The-Compiler

I personally think this is a bad idea as well, especially for tools which typically require multiple sections.

setup.cfg was originally intended as a configuration file for distutils/setuptools - why dump all the other unrelated stuff in there? What's wrong with having different files as separation for different tools? WIth the same rationale, you could argue it's better to have all your code in a single file, even if it's completely unrelated.

[obestwalter]

@bittner Do you have an opinion about if and how we should move this discussion forward here or was it moved to distutils-sig as @hpk42 suggested? I am trying to shorten our backlog and I would tend to close this issue here as it's scope extends just tox and is more about the Python testing ecosystem as a whole. So maybe this discussion should be led e.g. in distutils-sig (if it not already has happened ... I am not on that list (yet)).

[RonnyPfannschmidt]

@obestwalter PEP-518 introduced a file with a open namespace, however its not a ini file and currently the python parser available are not that good

[obestwalter]

To me this means we should close this issue here, because the discussion was already taken elsewhere, right?

[RonnyPfannschmidt]

@obestwalter the real issue as seen by @bittner and others is the massive overspill of config files in the project root ^^

just going pep-518 wint really help, since the structure of that file is contrary to basically all of tox
we could however obtain from it a different config file location

[obestwalter]

I still see this this discussion as being wildly out of scope for tox. This is a problem with the general ecosystem even exceeding the Python ecosystem. There is also massive diversity in tastes as well, as some seem to think it's nice to be able to configure tox/pytest/flake etc. inside a tox.ini or a setup.cfg and others think that's plainly wrong.

[RonnyPfannschmidt]

yes, from my pov the problem is not the number of files, but that everyone feels entitled to shit them into the root folder (i have been there and did the same)

[bittner]

I don't see that this problem is "exceeding the Python ecosystem". At least with giving the choice of placing configuration data for testing and packaging into (various) places would help on the Python side. We won't solve the JavaScript flood of config files, but that's not our job.

Personally, I've started to use tox.ini for putting config data into it, because this looks "semantically correct" for me when I use Tox for running just all my tests, including static code analysis (flake8, pylint, even jshint sometimes; see example).

I have a PR open with the behave project to make this come true for yet another tool. I think we should give the developers a choice. We don't have an isolated, clean approach to solve this problem domain anyway. Let's be pragmatic.

[RonnyPfannschmidt]

@bittner note that tox uses iniparse (which is vendored in pylib and incompatible with configparser and other parsers)

[warsaw]

I'd really like to see a move in the Python world to separating things needed to build/deploy/install the package and things needed to test it. I once thought python setup.py test would be the interface for that, but for several reasons that really didn't pan out. For a long while I've used, and in some ways has become a de facto standard, that tox is the standard way to invoke a project's test suite. That's nice!

It means to me that tox.ini is a natural place to put configurations for test-y things, like linters, coverage, etc. In fact, I've started to experiment with not even putting test-only dependencies in setup.py and only putting them in tox.ini. It's not a complete solution yet because I still have unittest.cfg files for things like nose2 plugins config.

[RonnyPfannschmidt]

@warsaw i like that idea, but i dislike the clutter because of lack of name-spacing

[hpk42]

Hum, not sure i can follow all comments here (also the bb-github move changed the order of some comments i guess). With @warsaw and @RonnyPfannschmidt strongly speaking against using setup.cfg for tox configuration and @obestwalter arguing the discussion is a bit out of scope i am inclined to close the issue for now. Hope that's ok for you, @bittner, as well.

It seems we all kind of agree that having pytest config (and maybe other testing tools config) in a tox.ini is less of a problem then putting it to setup.cfg. I still think it'd be great if small python projects could just have a single config file for packaging and testing but that requires looking at all current approaches and engaging in various discussions which i am not up to.

If discussing the involved issues further i suggest to bring them up on testing-in-python and/or distutils-sig.

[jleclanche]

@hpk42 Was this not done? Tox does pick up testenv from setup.cfg, it doesn't pick up tox section though.

[bittner]

@jleclanche @hpk42 @obestwalter As I said, we could give developers a choice. In my eyes the advantages (in "freedom") outweigh the (technical) disadvantages by far. It wouldn't be hard to allow setup.cfg as an additional, alternative configuration source.

Note that I'm saying this as someone who has switched to using tox.ini as a single source of configuration. I still support the idea though, in the light of educated freedom.

[RonnyPfannschmidt]

@bittner creating choices has maintenance cost, for choices that can be demonstrated as reasonably bad i dont see why a project should have to take the maintenance hit just because some want to engage in bad practices

[RonnyPfannschmidt]

also please consider that there is not just "freedom to" but also "freedom of" - and as volunteer driving projects the freedom from maintaining bad miss features is a massive win