Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Do not lint ignored file on stdin #7220

Merged
merged 5 commits into from
Sep 3, 2022
Merged

Do not lint ignored file on stdin #7220

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9f29f28
Test if ignored files passed on stdin are ignored
christoph-blessing Jul 22, 2022
6e35080
Do not lint ignored file when passed on stdin
christoph-blessing Jul 22, 2022
30d85df
Add news fragment for issue #4354
christoph-blessing Jul 22, 2022
68ea2d5
Merge branch 'main' of https://github.com/PyCQA/pylint into issue-4354
DanielNoord Sep 2, 2022
9da9b5d
Small refactor
DanielNoord Sep 2, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions doc/whatsnew/fragments/4354.bugfix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Fix ignored files being linted when passed on stdin.

Closes #4354
7 changes: 7 additions & 0 deletions pylint/lint/pylinter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -641,6 +641,13 @@ def check(self, files_or_modules: Sequence[str] | str) -> None:

filepath = files_or_modules[0]
with fix_import_path(files_or_modules):
if _is_ignored_file(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we also remove the previous place where we where ignoring some file ? If this is the right place then we don't need to check in the other places it was checked ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not completely sure I understand. The check was not present previously in this if statement. Do you mean that we should put the check before the second if statement and remove it from all the subsequent if statements?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're already filtering files somewhere when they are not from stdin, I think there's a kind of refactor to do there so we filter files at the right place and so it's efficient (done only once per run, possibly when using multiprocessing). I would start by checking where _is_ignored_file is used.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That function is used in two places:

It seems like the expand_modules function is the place where the filtering of ignored files is meant to take place and the use of _is_ignored_file while discovering is more of a "hack". Unfortunately expand_modules does not get called when linting from standard input. Furthermore expand_modules seems to do two things:

  1. Expand modules
  2. Filter ignored files

It might be a good idea to entangle these two concerns. In any case I don't see a way to only filter once. Before expand_modules you want to filter once to avoid unnecessarily processing any ignored files but you can not filter everything at this stage because you do not have filepaths for the modules yet. Therefore you want to filter again after you have those.

So I am not sure how to proceed from here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for checking this.

It might be a good idea to entangle these two concerns. In any case I don't see a way to only filter once. Before expand_modules you want to filter once to avoid unnecessarily processing any ignored files but you can not filter everything at this stage because you do not have filepaths for the modules yet. Therefore you want to filter again after you have those.

Hmm, it sounds like a bigger refactor than I thought. If we're going to extract the file discovering from PyLinter we could expand modules, reading from stdin and files discovery at the same time, right ? Removing concerns from the PyLinter would be a good thing especially for more efficient multiprocessing (easier to map files to threads), but I don't have a clear vision about the issue we'll face when doing that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think it should be possible to do the discovering, filtering and expanding beforehand and then passing the result to Pylinter.check but as you said that is going to be a larger refactor.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you be willing to give it a try ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to give it a try. That said I have no idea how long it will take because it seems like a quite big refactor and I do not have a lot of time to work on this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice to hear, don't worry about deadline there is none :) ! You can open a draft pull request to discuss the refactor with maintainers, early feedback would probably help you.

filepath,
self.config.ignore,
self.config.ignore_patterns,
self.config.ignore_paths,
):
return
self._check_files(
functools.partial(self.get_ast, data=_read_stdin()),
[self._get_file_descr_from_stdin(filepath)],
Expand Down
14 changes: 14 additions & 0 deletions tests/test_self.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1113,6 +1113,20 @@ def test_ignore_pattern_recursive(self, ignore_pattern_value: str) -> None:
code=0,
)

def test_ignore_pattern_from_stdin(self) -> None:
"""Test if linter ignores standard input if the filename matches the ignore pattern."""
with mock.patch("pylint.lint.pylinter._read_stdin", return_value="import os\n"):
self._runtest(
[
"--from-stdin",
"mymodule.py",
"--disable=all",
"--enable=unused-import",
"--ignore-patterns=mymodule.py",
],
code=0,
)

@pytest.mark.parametrize("ignore_path_value", [".*ignored.*", ".*failing.*"])
def test_ignore_path_recursive(self, ignore_path_value: str) -> None:
"""Tests recursive run of linter ignoring directory using --ignore-path parameter.
Expand Down