No output from flake8 on Windows

[deadkex]

Hi!
I am using darker in combination with pre-commit and the linters i would like to use are pylint and flake8.
The problem is that i cannot get any output from flake8, on the other hand pylint works as expected.
If you take below config and pre-commit run --file test.py, you can see that flake8 is actually running and reporting violations, but darker is not showing them. I also tried committing larger files with lots of changes or explicitly adding violations as changes.

python: 3.8.10
darker: 1.5.0
pre-commit: 2.19.0
flake8: 4.0.1
black: 22.3.0

.pre-commit-config.yaml

default_language_version:
    python: python3

repos:
-   repo: local
    hooks:
    -   id: darker
        name: darker
        entry: darker
        language: system
        types: [python]
        args: [
            "-L flake8 -v"
        ]
    -   id: flake8
        name: flake8
        entry: flake8
        language: system
        types: [ python ]

pyproject.toml

[tool.darker]
src = ["./"]
revision = ":PRE-COMMIT:"
diff = false
check = true
isort = false
# lint = [
#     "flake8"
# ]

[tool.black]
target-version = ["py38"]
include = '\.py?$'

test.py

try:
    pass
except:
    pass

[akaihola]

Hi @deadkex, thanks for the detailed problem report!

I started by trying to reproduce this without pre-commit, but at least Darker works correctly here:

mkdir /tmp/drktst
cd /tmp/drktst
git init
cat <<EOF >test.py
try:
    x = 1
except Exception:
    print("exception")
EOF
git add test.py
git commit -m "Initial commit"
cat <<EOF >test.py
try:
    pass
except:
    pass
EOF
darker -L flake8 .

/tmp/drktst/test.py:3:1: E722 do not use bare 'except'
/tmp/drktst/test.py:3:1: B001 Do not use bare `except:`, it also catches unexpected events like memory errors, interrupts, system exit, and so on.  Prefer `except Exception:`.  If you're sure what you're doing, be explicit and write `except BaseException:`.

So probably this is a problem in either how Darker behaves when run by pre-commit, or in the pre-commit configuration.

Does this help you forward in diagnosing the issue?

[deadkex]

Hello @akaihola
I forgot to add that i am working on a Windows pc.

Having that in mind, i tried to replicate this on a Linux VM and the project and also the test you sent worked fine.

On Windows:

1. Using darker with your test
   I don't get any output for flake8
   

2. Using darker in our project
   I get a "WindowsError: [Error 206] The filename or extension is too long". When running flake8 with pre-commit in this project, it works fine.
   

If there's anything you want me to test, feel free to ask.

[akaihola]

Interesting!

First thing that comes to mind – is the output of flake8 identical on Linux and Windows? I mean, if you run flake8 test.py, does it output the file paths and line numbers identically? What about directory separators?

To debug your FileNotFoundError, you could run

python -m pdb C:\Python38\Scripts\darker.exe\__main__.py -L flake8 .

and when it stops at the exception, execute the up command 4 times so you get to git.py:243, then see what the values of cmd and str(cwd) look like.

[deadkex]

Hey, sorry for the late reply.
Windows python: 3.8.10
Linux python: 3.8.8

The output of flake8 test.py on Windows and Linux - seems to be the same

Here is the output of darker -L flake8 test.py on win/linux

Darker just doesn't want to print the output of flake8, but seems to detect that something is wrong.

This can be confirmed when running darker via pre-commit - it fails but it just doesn't print flake8's output:
precommit -> darker with flake8:

precommit -> darker with pylint:

Edit: pre-commit + darker + flake8 work fine in the project when used on linux

---

Now about the WindowsError:
Just fyi to debug darker on windows, i had to use
python3 -m pdb C:\Python38\lib\site-packages\darker\__main__.py -L flake8 .
This can be seen in the next picture (variable cmd: 1530 elements / 93287 chars from which ~800 elements are from pyinstaller)

Debugged on Linux (added a breakpoint in git.py:243)

Edit: The WindowsError only appears when running the command in the console, i don't get it when running via pre-commit which means it doesn't really affect me

[akaihola]

I added a bit of debug output when running linters, see #373. With that patch, if I run Darker on its own codebase, I see this:

$ darker -vv -L flake8 .
[...]
DEBUG:darker.linting:[/home/akaihola/darker]$ flake8 /home/akaihola/darker/src/darker/linting.py

We can see that Darker likes to always use full absolute paths for files to lint. Can you try and see what the debug output looks like for you if you install Darker from #373, i.e. the linter-log-output branch?

Then, based on that, you could take a look at flake8 output on Linux and Windows (substituting correct paths):

# Linux
python3.8 -m flake8 ~/PycharmProjects/tmp/test.py

# Windows
python3.8 -m flake8 C:\Users\deadkex\Documents\tmp\test.py

I also suspect that WindowsError: [Error 206] The filename or extension is too long is due to the full path. Do you know what the maximum length of a file path on Windows is?

[deadkex]

Hey, i encountered another thing.
I tried upgrading to Python 3.10 but now i get a new error, which comes when using pylints config file option.
The file is freshly generated with pylint --generate-rcfile > pylint.conf

lint = [ "pylint --rcfile=tools/pylint.conf" ]

[WARNING] Unstaged files detected.
[INFO] Stashing unstaged files to ...\.cache\pre-commit\patch1660718692-12008.
darker...................................................................Failed
- hook id: darker
- exit code: 1
Traceback (most recent call last):
  File "C:\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Python310\Scripts\darker.EXE\__main__.py", line 7, in <module>
  File "C:\Python310\lib\site-packages\darker\__main__.py", line 487, in main_with_error_handling
    return main()
  File "C:\Python310\lib\site-packages\darker\__main__.py", line 469, in main
    linter_failures_on_modified_lines = run_linters(
  File "C:\Python310\lib\site-packages\darker\linting.py", line 173, in run_linters
    return sum(
  File "C:\Python310\lib\site-packages\darker\linting.py", line 174, in <genexpr>
    run_linter(linter_cmdline, root, paths, revrange, use_color)
  File "C:\Python310\lib\site-packages\darker\linting.py", line 133, in run_linter
    edited_linenums = edited_linenums_differ.compare_revisions(
  File "C:\Python310\lib\site-packages\darker\git.py", line 460, in compare_revisions
    return _compare_revisions(
  File "C:\Python310\lib\site-packages\darker\git.py", line 429, in _compare_revisions
    linenums = _revision_vs_lines(root, path_in_repo, rev1, content, context_lines)
  File "C:\Python310\lib\site-packages\darker\git.py", line 402, in _revision_vs_lines
    multiline_string_ranges = get_multiline_string_ranges(content)
  File "C:\Python310\lib\site-packages\darker\multiline_strings.py", line 20, in get_multiline_string_ranges
    return [
  File "C:\Python310\lib\site-packages\darker\multiline_strings.py", line 20, in <listcomp>
    return [
  File "C:\Python310\lib\tokenize.py", line 523, in _tokenize
    raise TokenError("EOF in multi-line statement", (lnum, 0))
tokenize.TokenError: ('EOF in multi-line statement', (283, 0))
[INFO] Restored changes from ...\.cache\pre-commit\patch1660718692-12008.

[akaihola]

I created GitHub issues #374 and #375 for the problems reported here. Closing the discussion, to be continued in the issues. Thanks @deadkex!