Add support for additional predefined config file paths #2761

miedzinski · 2017-01-26T21:15:21Z

When --config-file option isn't passed to mypy and mypy can't load mypy.ini (default config filename), it will try to load settings from common shared config files -- currently setup.cfg and tox.ini. However, with these files, mypy will become less verbose; specifically it won't require mypy section.

My main incentive is that I don't wan't to "bloat" my project structure with loads of config files; OTOH I expect them to work just by typing well-known command (with options loaded from configuration files). Two common development tools I use are pytest and flake8 and both support loading setup.cfg files. Another popular one is tox, which unfortunately doesn't load that file, but the former two load tox.ini in addition to setup.cfg. Moreover, tox is often used to run all these tools with mypy included and I believe some people consider keeping all configuration in one place as a nice thing to have.

I do understand that this is kind of an anti-pattern, but with small codebases it is quite annoying to have as many config files as source code. This is why it is implemented as a completely optional feature, with --config-file option and mypy.ini taking precedence over shared configs. Anyway, this PR is rather small - I hope the code won't need any maintenance - and some people will benefit from it, while others will never use it.

ddfisher · 2017-01-27T00:32:39Z

Hi! Thanks for submitting this PR.

If I recall correctly, when implementing config files initially, we considered using setup.cfg and tox.ini as fallbacks and decided against it. I don't remember the rationale, though -- @gvanrossum, do you remember?

gvanrossum · 2017-01-27T01:58:38Z

I prefer not to do this. To explain my reasoning (then and now): as a consumer of projects created by others I've often been confused by the rules for finding configuration data in seemingly randomly-named files like tox.ini (I've never heard of anyone who uses it) and setup.cfg, combined with directory searches. E.g. if there's a tox.ini in a subdirectory but a setup.cfg in the root directory of a project, which one wins, or are they combined somehow?

miedzinski · 2017-01-27T12:41:15Z

@ddfisher, @gvanrossum,

I understand and this is something I've expected. I think my reasoning can be explained when we look at hypothetical open-source Python project, which decides to incorporate quite a few modern tools:

flake8 - code linting
pytest - test framework
coverage.py - code coverage analysis
mypy - type checking
tox - virtualenv management and test runner combined (by the way it is quite common; e.g. python/asyncio uses it)

This project may also use other tools, which require configuration, but we'll ignore them (for instance it may use something like isort, probably have requirements.txt file, maybe requirements-(dev|test|docs).txt or it uses brand new Pipfile and Pipfile.lock).

Now let's say some new developer decides to fork the repo and hack some things on it. Sooner or later, he will want to test/lint/typecheck/something his code. The project probably has some kind of runtests.py script; since it uses tox running all suites may be as simple as typing tox. However, the developer often wants to just lint his code or typecheck it and he knows that flake8 and mypy do that. In his first try he will try typing these commands (some may try something like tox -e flake8,mypy because it has tox). How can we make sure this will work as expected? By having these tools configured and making sure they pick up configs out of the box (without explicitly passing them as CLI option).

What will be read?

flake8 - .flake8, setup.cfg, tox.ini, and config/flake8 on Windows
pytest - pytest.ini, tox.ini, setup.cfg
coverage.py - .coveragerc, setup.cfg, tox.ini
mypy - mypy.ini
tox - tox.ini

One file per tool or shared configs. It's possible to stick everything into tox.ini or split into tox.ini and setup.cfg - tox will probably run these tools, which will all read setup.cfg. Often, many projects decide to have dedicated files for some tools, and shared for others (mixing conventions). Take a look at Django: setup.cfg contains flake8 and isort (it uses .isort.cfg by default), .coveragerc (in tests sub-project) and of course tox.ini.

Mypy devs themselves have decided that flake8 and coverage.py can have their configs in setup.cfg, while keeping pytest in pytest.ini. In some sense it is unfair that mypy itself "requires" dedicated file.

Personally, I see (at least) two types of tools:

build tools (setuptools - setup.py, pip - requirements.txt, pipenv - Pipfile[.lock])
code analysis (flake8, pytest, mypy included)

In the former, I believe they should have only dedicated files (and they do), because these are required to turn source code into a program. The latter should not, since they are only to help and it should be up to developers how to configure them. Since there is visible trend in the community to use setup.cfg, we should do the same.

Why I don't want that many files? Try to write a modern (defined as using FOTM frameworks and build tools) JS app and you'll see. Even django, despite being mostly Python codebase, has four JS-related configs in project root. A lot of people say JS ecosystem has some flaws in it, and, IMO, configs scattered all over the place contribute to it. Again, in my opinion, if an user wants to have mypy configured out of the box in his codebase, we shouldn't force him to create yet another file. If developers want to keep all configs in single place, noone will stop them and they will work around that in either way, e.g. telling contributors to run some tools with --config-file setup.cfg option.

Lastly, regarding subdirectories and merging configs. AFAIK none of these tools do this and neither would mypy after merging this PR. It would look for --config-file option and then, assuming user didn't specify CLI option, read well-known files in project root in order:

mypy.ini
setup.cfg
tox.ini - to be honest I don't like this one and was thinking on whether it should be read; decided to add it because other tools do so

If mypy manages to read some file, it stops reading next. This why there won't be any file merging; I think noone does it and it'd be a hell of a nightmare to manage. Additionally - for already configured projects - if you have mypy.ini and/or no mypy related entries in shared configs, mypy won't change its configuration.

EDIT:
I've misunderstood Guido in regard to tox. I thought you don't know anyone using tox - pardon me. Like I said above, I agree that tox.ini may not be the best file to read config from and I could accept dropping it from this PR.

gvanrossum · 2017-01-27T15:52:22Z

I am still not convinced. The desire for fewer files in the project root is honest, but naive. There are going to be tons of files there, like it or not. It's just how the world works. And the rules for precedence when multiple of these files exist are very murky (I know for a fact that a setup.cfg in a subdirectory overrides one in a parent directory if you point pytest to files in that subdirectory).

habnabit · 2017-01-27T18:49:23Z

I am still not convinced. The desire for fewer files in the project root is honest, but naive. There are going to be tons of files there, like it or not. It's just how the world works.

Given a chance to try to reverse this trend, why not take it?

gvanrossum · 2017-01-27T19:01:56Z

Given a chance to try to reverse this trend, why not take it?

See my earlier reply. The multiple files are confusing.

miedzinski · 2017-01-27T19:32:51Z

@gvanrossum, please consider reopening. The actual discussion has just begun to appear. I've brought up a topic on Python tools about using setup.cfg on freenode's #python IRC channel and, from what I see, people seem to be happy about it.

I'd be cautious when saying there will be tons of files anyway. I am using pytest, flake8, and mypy in my project and with this PR I can halve config files amount - to one.

The rules for precedence are easy:

read --config-file option - if it's incorrect, exit
read mypy.ini - if correct, stop
read setup.cfg

Subdirectory behavior is clear as well - mypy doesn't traverse directory tree looking for mypy.ini and it wouldn't do so to find setup.cfg. Pytest behavior is irrelevant here.

By the way, would you close PR with support for PEP 512? Like it or not, the community (at least some part of it) has agreed on setup.cfg. Until PEP 512 becomes widely adopted, there is nothing you can do. I really dislike that mypy wants to be a special snowflake and force users to create a dedicated file, usually with only few lines in it.

And please don't say that setup.cfg would be confusing. That's nonsense. If an user writes mypy options there, it won't be surprising if mypy reads them. I'm pretty sure a lot of Python developers know about setup.cfg purpose, because they configure a lot of tools there. Remember, noone forces anyone to use it - mypy.ini should be preferred, and with this PR, it is.

Additionally, after @nedbat suggestion, I've dropped tox.ini support. This is something I agree with, but was included in just because other tools do so.

EDIT:
Here is the tox.ini commit - miedzinski/mypy@3240219. Looks like GitHub won't pick it up until reopened.

gvanrossum · 2017-01-27T20:02:16Z

I'll reopen, but I will continue to argue against. Look at https://github.com/python/mypy, the toplevel has already 18 different files, a majority of which are configuration of some sort or other.

Maybe @ambv has an opinion? He's got more experience with a variety of tools like this than I have. (And I notice that he added a .flake8 file to typeshed rather than a setup.cfg.)

gvanrossum · 2017-01-27T20:03:19Z

docs/source/command_line.rst

-  and command line flags can override settings.  See :ref:`config-file`
-  for the syntax of configuration files.
+  read from the given file. By default settings are read from ``mypy.ini``,
+  ``setup.cfg``, or ``tox.ini`` in the current directory. Settings override


You still mention tox.ini here (and several other places). Also you should clarify that it stops at the first file it finds (here and in the next chapter).

illume · 2017-02-07T18:24:51Z

I see flake8 and converage config are in the setup.cfg file now at https://github.com/python/mypy

Note that pytest can also use setup.cfg (and tox.ini). See how they do it here: http://doc.pytest.org/en/latest/customize.html

Some packages are really drowning in config file soup, with 20-30 config files! Hopefully at least all the python tools can agree on a common file for config.

ambv · 2017-02-13T06:33:57Z

Sorry, didn't notice the @ until now. I added a .flake8 rather than a setup.cfg because there's no setup.py so using setup.cfg felt wrong.

That being said, I don't think additional support for setup.cfg configuration would be confusing. I imagine that the general rules would be that:

--config-file= trumps all
within a directory, mypy.ini trumps setup.cfg
no overriding of config in sub-directories
no combining of multiple sources of config

With those explicitly stated in the docs, I think there's little space for surprise on the user's end.

Oh, and I agree that tox.ini can be ignored.

gvanrossum · 2017-02-13T15:18:21Z

OK, you've convinced me. Is that the state of the PR currently? Then I will merge it.

miedzinski · 2017-02-13T15:51:25Z

It is exactly as @ambv described.

gvanrossum

Bunch of style nits. We're almost there!

gvanrossum · 2017-02-13T17:22:38Z

docs/source/command_line.rst

@@ -344,10 +344,10 @@ Here are some more useful flags:
 .. _config-file-flag:

 - ``--config-file CONFIG_FILE`` causes configuration settings to be
-  read from the given file.  By default settings are read from ``mypy.ini``


Please put the double spaces after periods back. They don't affect how the HTML is rendered, but they improve the quality of paragraph reflowing in Emacs.

gvanrossum · 2017-02-13T17:23:03Z

docs/source/command_line.rst

-  and command line flags can override settings.  See :ref:`config-file`
-  for the syntax of configuration files.
+  read from the given file. By default settings are read from ``mypy.ini``
+  and ``setup.cfg`` in the current directory. Settings override mypy's


maybe "and" -> "or" ?

gvanrossum · 2017-02-13T17:23:51Z

docs/source/config_file.rst

+read a different file instead (see :ref:`--config-file <config-file-flag>`).
+
+It is important to understand that there is no merging of configuration
+files, as it would lead to disambiguity. The ``--config-file`` flag


I'm not sure if I understood this correctly. You meant first sentence only, right?

I actually just meant to change "disambiguity" to "ambiguity". What you wrote was backwards. :-)

Oh, you're absolutely right. Restored the sentence.

gvanrossum · 2017-02-13T17:25:02Z

mypy/main.py

    """Parse a config file into an Options object.

    Errors are written to stderr but are not fatal.
+
+    If filename isn't provided, fall back to default config file and then


"isn't provided" -> "is None"

gvanrossum · 2017-02-13T17:25:42Z

mypy/main.py

    """Parse a config file into an Options object.

    Errors are written to stderr but are not fatal.
+
+    If filename isn't provided, fall back to default config file and then
+    to common shared files (currently setup.cfg).


I would just say "to setup.cfg". If and when we change it we can update the comment.

gvanrossum · 2017-02-13T17:27:39Z

mypy/main.py

+    if filename is not None:
+        config_files = (filename, )
+    else:
+        config_files = (defaults.CONFIG_FILE, ) + SHARED_CONFIG_FILES


No space between "," and ")". (Also two lines up.)

gvanrossum · 2017-02-13T17:28:24Z

mypy/main.py

@@ -540,31 +537,52 @@ def get_init_file(dir: str) -> Optional[str]:
    'almost_silent': bool,
 }

+SHARED_CONFIG_FILES = ('setup.cfg', )


No space between "," and ")".

miedzinski · 2017-02-13T20:08:55Z

Fixed. Please forgive my English - I'm trying to do my best despite it's not my native language. And I'm more of a Vim guy. :)

When --config-file option isn't passed to mypy and mypy can't load mypy.ini (default config filename), it will try to load settings from setup.cfg.

gvanrossum · 2017-02-14T18:35:33Z

Thanks!

PS. Next time can you not squash your commits? It's easier to go through review cycles if every push is a separate commit. e squash and rewrite the commit message when we merge PRs anyways.

gvanrossum closed this Jan 27, 2017

gvanrossum reopened this Jan 27, 2017

gvanrossum reviewed Jan 27, 2017

View reviewed changes

miedzinski force-pushed the setupcfg branch from 3240219 to 0e9ca39 Compare January 27, 2017 20:54

gvanrossum requested changes Feb 13, 2017

View reviewed changes

miedzinski force-pushed the setupcfg branch from 0e9ca39 to fe44589 Compare February 13, 2017 20:05

Read configuration from setup.cfg by default

f8535b1

When --config-file option isn't passed to mypy and mypy can't load mypy.ini (default config filename), it will try to load settings from setup.cfg.

miedzinski force-pushed the setupcfg branch from fe44589 to f8535b1 Compare February 14, 2017 11:46

gvanrossum approved these changes Feb 14, 2017

View reviewed changes

gvanrossum merged commit 70b278c into python:master Feb 14, 2017

matrixik mentioned this pull request Oct 26, 2017

Search pylint configuration in setup.cfg and pyproject.toml pylint-dev/pylint#617

Closed

Flimm mentioned this pull request Nov 22, 2017

Create a tox.ini file jazzband/sorl-thumbnail#525

Closed

OJFord mentioned this pull request Sep 19, 2019

No longer respects [mypy] section in tox.ini? #7532

Closed

redeboer mentioned this pull request Apr 28, 2020

ci: add mypy configuration and pre-commit hook ComPWA/tensorwaves#48

Merged

dnshio mentioned this pull request Oct 5, 2020

Add mypy sqlfluff/sqlfluff#459

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for additional predefined config file paths #2761

Add support for additional predefined config file paths #2761

miedzinski commented Jan 26, 2017

ddfisher commented Jan 27, 2017

gvanrossum commented Jan 27, 2017

miedzinski commented Jan 27, 2017 •

edited

Loading

gvanrossum commented Jan 27, 2017 via email

habnabit commented Jan 27, 2017

gvanrossum commented Jan 27, 2017

miedzinski commented Jan 27, 2017 •

edited

Loading

gvanrossum commented Jan 27, 2017

gvanrossum Jan 27, 2017

illume commented Feb 7, 2017

ambv commented Feb 13, 2017

gvanrossum commented Feb 13, 2017 via email

miedzinski commented Feb 13, 2017

gvanrossum left a comment

gvanrossum Feb 13, 2017

gvanrossum Feb 13, 2017

gvanrossum Feb 13, 2017

miedzinski Feb 13, 2017

gvanrossum Feb 13, 2017

miedzinski Feb 14, 2017

gvanrossum Feb 13, 2017

gvanrossum Feb 13, 2017

gvanrossum Feb 13, 2017

gvanrossum Feb 13, 2017

miedzinski commented Feb 13, 2017

gvanrossum commented Feb 14, 2017

Add support for additional predefined config file paths #2761

Add support for additional predefined config file paths #2761

Conversation

miedzinski commented Jan 26, 2017

ddfisher commented Jan 27, 2017

gvanrossum commented Jan 27, 2017

miedzinski commented Jan 27, 2017 • edited Loading

gvanrossum commented Jan 27, 2017 via email

habnabit commented Jan 27, 2017

gvanrossum commented Jan 27, 2017

miedzinski commented Jan 27, 2017 • edited Loading

gvanrossum commented Jan 27, 2017

Choose a reason for hiding this comment

illume commented Feb 7, 2017

ambv commented Feb 13, 2017

gvanrossum commented Feb 13, 2017 via email

miedzinski commented Feb 13, 2017

gvanrossum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

miedzinski commented Feb 13, 2017

gvanrossum commented Feb 14, 2017

miedzinski commented Jan 27, 2017 •

edited

Loading

miedzinski commented Jan 27, 2017 •

edited

Loading