lint all files in a directory

[pylint-bot]

Originally reported by: Buck Evan (BitBucket: bukzor, GitHub: @bukzor?)

---

It would make things easier for me to use pylint in CI if I could simply say pylint . in order to tell pylint to look at all my files.

flake8/pep8 already have this behavior. They make it configurable via a filepatterns config (default *.py) and an exclude config (default .svn,CVS,.bzr,.hg,.git,__pycache__).

Discussion from #pylint:

buck1: jcristau: in flake8, I can run `flake8 .` and it has a set of configured match/exclude patterns to check everything in the directory
buck1: currently pylint assumes that any directory is a package, and fails if not
buck1: how would you feel about including the flake8 behavior
buck1: would make my life 100% easier when trying to run pylint against everything during CI
jcristau: doesn't sound crazy, though there might be backwards compat concerns...  please float the idea in a ticket or on the list, i'm not the maintainer :)
buck1: jcristau: i'll make a feature-request ticket. Sounds like something I might be able to implement. Just wanted to see your thoughts first.

---

• Bitbucket: https://bitbucket.org/logilab/pylint/issue/352

[pylint-bot]

Original comment by Claudiu Popa (BitBucket: PCManticore, GitHub: @PCManticore):

---

Sounds good. I also encountered the need for this feature when running pylint over a large corpora of scripts. Nevertheless, Pylint shouldn't emit every time when it finds a folder without an init file, as in the case of the namespace packages from Python 3.3+.

[pylint-bot]

Original comment by Claudiu Popa (BitBucket: PCManticore, GitHub: @PCManticore):

---

Issue #512 was marked as a duplicate of this issue.

[pylint-bot]

Original comment by BitBucket: rneu31, GitHub: @rneu31?:

---

Would the entire directory be scanned in a serial manner (in one thread/process)?

For the CI process where I work we wrote a wrapper script that uses multiprocessing and consolidates the outputs of pylint (and various other tools) to avoid the large time hit of running Pylint across a large set of files.

[pylint-bot]

Original comment by Claudiu Popa (BitBucket: PCManticore, GitHub: @PCManticore):

---

The scanning should be serial, that is finding the files. The actual processing can already be parallelized with --jobs. So, as long as this https://bitbucket.org/logilab/pylint/src/33e334be064caf5b5d251929fc2f74bdf400f6c4/pylint/lint.py?at=default#cl-882 returns all the files from a directory that should be analyzed, then the parallelization comes freely with --jobs.

[pylint-bot]

Original comment by Florian Bruhin (BitBucket: The-Compiler, GitHub: @The-Compiler?):

---

@PCManticore mentioned someone worked on this on the EuroPython sprints but never uploaded a PR - if you're that person, can you maybe upload a PR (even if work-in-progress), so I could look at this when I have some time?

[rohsaini]

I am looking into this enhancement. Will provide the update once code is ready.

[rohsaini]

Hi,
Can anyone help me here to understand why CI might have stalled or failed?

Thanks,
Rohit

[rneu31]

Is there any update on this / will this be a part of the 2.0 release?

[PCManticore]

No one continued work on it. Unless someone takes charge of it, it will probably not be part of the 2.0 release. I will probably not have time to tackle it, since I have other priorities to bring to an end for 2.0.

Note that in 2.0 we support implicit namespace packages. which have no init.py inside and supporting generic directories with scripts should not be that hard after all.

[rneu31]

I am unfortunately stuck using 2.7. I have been following some of the discussions on work being done for 2.0, but not the code directly. Is there any work in this area that makes sense to do?

[PCManticore]

Yes, implementing this should be pretty straightforward once you figure out the details.

I will provide some high level details, in case you might be interested in tackling this (or if someone else see the comment as well):

• this behavior should be triggered through a flag. It can either default to False, which means the current behavior of not processing directories without __init__.py or we can default it to true, so it would act more in line with the other tools, but in any way, it has to be controllable through a flag.
• We are collecting the files for analysis using this function https://github.com/PyCQA/pylint/blob/master/pylint/utils.py#L800. It uses inside another function https://github.com/PyCQA/astroid/blob/master/astroid/modutils.py#L460. The list_all flag of modutils.get_module_files can be used for listing all the files, indifferent of having or not __init__.py in that directory.
• From here, tying these things together should be straightforward, including having tests and whatnot.

[lindsaymarkward]

I would like to add that I'm keen on this proposed functionality - pylint .

[jamesqo]

Even better-- just pylint should have the same behavior as pylint ..

[razzeee]

Would be really awesome to get this fixed

[PCManticore]

Hey folks, just wanted to say that I left a couple of suggestions before on how this feature could be implemented in pylint. As previously mentioned, we'll get to it someday, but as far as it concerns me, I don't have enough time on my plate for implementing this feature. A PR from someone else could definitely speed this up!

[razzeee]

As we do depend on this a little bit it would be awesome to get it tackled and I would even enable our gsoc student to have a look at it. What would be helpful would be any insight or mentorship he could get, as he obviously doesn't know the codebase and it seems to be quiet a deep change?

[PCManticore]

Hey @razzeee That sounds like a great first step. I can offer mentorship on this issue if someone wants to write the code for it. I suggested an initial step in this comment #352 (comment) but I would be happy to write a more in-depth comment on what exactly needs to be changed.

[mzfr]

@PCManticore After looking into this and following the comment I was able to figure it out.

In astroid/modutils.py - L472 we are stoping the get_module_files function on 2 conditions

• If list_all is not True
• If there is no __init__.py file present in the given directory.

E.g
Example
├── File.py
├── file.txt
└── Folder
    └── new.txt

Here src_directory = "Example"
Now doing get_module_files(src_directory, blacklist) would return an empty list
and doing get_module_files(src_directory, blacklist, True) would return a list having path to all the files i.e ['Example/file.txt', 'Example/File.py', 'Example/Folder/new.txt']

Solution:
We can't just remove the __init__.py condition because we actually want this to be a command line flag so for that we can modify the code.

def get_module_files(src_directory, blacklist, list_all=False, init=False):
   files = []
   for directory, dirnames, filenames in os.walk(src_directory):
       if directory in blacklist:
           continue
       _handle_blacklist(blacklist, dirnames, filenames)
       if not list_all and not init:
           dirnames[:] = ()
           continue
       for filename in filenames:
           if _is_python_file(filename):
               src = os.path.join(directory, filename)
               files.append(src)
   return files

Here init=False means behavior of not processing directories without __init__.py

After this modification

• get_module_files(src_directory, blacklist) would again return an empty list
• get_module_files(src_directory, blacklist, True) or get_module_files(src_directory, blacklist,False, True) would give list of all the files.

Please tell me whether I am on right track or not.

Now the thing I was not able to understand was your refrence to pylint/utils - L800. What will I have to do there ?

Also can you tell me which file will I have to change to make this a command line flag

[PCManticore]

Hey @mzfr That sounds about right! Thanks for looking into this! Regarding the reference from utils.py:800, I have no idea! The code has changed since I last linked it. Regarding the file to change, you can take a look into lint.py, if you look for options you should find the place where it could go, Out of the top of my head, I'd say that in either Runner or PyLinter. Let me know if this helps!

[mzfr]

@PCManticore ok so I am going to ignore utils:800 now.
For options I think this is where I have to add the options.
So following the format the flag would look like:

(('init',
  {'default' : 'True','type': 'yn' 'metavar' : '<y_or_n>',
   'help' : 'When set to False - not processing directories without __init__.py'
            'when set to True  - processing directories without __init__.py'}),

This is what I think let me know whether should I add any short or group in the option.

[razzeee]

This is what @PCManticore linked to back when he referenced to that line https://github.com/PyCQA/pylint/blob/5d40a07684be7ed6dbaa54e49c3d436005a533bf/pylint/utils.py#L800

[PCManticore]

@mzfr Sounds about right! One thing though, init as a name doesn't suggest the intention but rather the semantics. Maybe lint-all or something along these lines?

[dqal]

I think this issue may also prevent pylint from being used in pyproject.toml files as there is no flag that can be passed in the following way to specify directories.

black style (RegEx string)

[tool.pylint]
include = '(lib|test)/.+\.py$'

isort style (list of dirs)

[tool.pylint]
src_paths = [
    "lib",
    "test",
]

[felixvd]

Is there a way to contribute to a bounty or something? I would happy to buy a pizza for whoever implements this.

[Pierre-Sassoulas]

You can create a bounty on issue hunt for example. Maybe what you refer to as a 'pizza' is equivalent to multiple days of work in another country than the one you live in. :) If pylint is not set up for Issue Hunt let me know, I'll do it.

[tuukkamustonen]

I've started to use find (shell command) to collect the files for all of the static analysis tools (pycodestyle, pydocstyle, mypy, pylint). We normally want to feed the same files for each anyway, but they each pretty much have a different way to collect the files (different args and functionalities how to include or exclude files, different defaults, etc).

A simple example:

pylint $(find . -iname "*.py")

You of course cannot put this file discovery in setup.cfg or pylint.cfg, so it's not a good workaround (unless you have Makefile or similar to drop it in)...

[Pierre-Sassoulas]

Thanks to @matusvalo this will be available in pylint 2.13.0. There's a follow up discussion concerning the possibility to launch pylint without argument in the current directory in #5701, please cast your vote if you have an opinion on the issue :)

[matusvalo]

Is there a way to contribute to a bounty or something? I would happy to buy a pizza for whoever implements this.

Now I will wait for pizza :-D (just kidding)

[felixvd]

Thanks!

Tell me how you would like to receive a coupon or money and it will be done. I can do SEPA transfers. [email protected]

[Pierre-Sassoulas]

@felixvd If you mean it, you can use github sponsor : https://github.com/sponsors/matusvalo 😄 It's matched by github the first year so it might very well even be two (2) pizzas in the end 🎉

[matusvalo]

@felixvd If you mean it, you can use github sponsor : https://github.com/sponsors/matusvalo 😄 It's matched by github the first year so it might very well even be two (2) pizzas in the end 🎉

Or even better donate to pylint project or maintainers of pylint. The money will be better spent for you if you are using pylint. My message was pure joke. I could not resist to post it :-D

[felixvd]

It's matched by github the first year so it might very well even be two (2) pizzas in the end

The more you know! I hadn't heard of that, guess it's worth using.

I sent you half of @matusvalo 's pizza fee, but his sponsor page does not have a one-time contribution option. @matusvalo maybe you could set that up?

[felixvd]

@matusvalo I tried this out, the option is kind of hidden in Sponsor Tiers/Custom amounts/Enable one-time custom amounts. Surprised that it's not on by default. If you turn it on I'll send the half-pizza.

[DanielNoord]

You need to set up a connected bank account with Stripe Express, which is probably the reason why it's not turned on by default 😉

Looking at @matusvalo's profile, they have turned it on though 😄

[matusvalo]

@felixvd I prefer not to enable one-time payments (I have some practical reasons for that). If you still want to do one-time contribution, I have sent you my IBAN over the e-mail ([email protected]).