Same error is reported multiple times when using multiprocessing (including performance decrease)

[robsan00]

Steps to reproduce

1. create a new virtualenv with latest pylint development state (3fffdf0)
2. create a folder "test" somewhere
3. add an (empty) "init.py" file there
4. add an (empty "spam1.py" file there, write "asdf" and nothing else in that file
5. copy the file 20 times in this same folder, calling the files "spam2.py" etc.
6. go to the parent directory of "test" and call "pylint -j 1 -E test":

(pylint) C:\work\pylint_test>pylint -j 1 -E test
************* Module test.spam1
test\spam1.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam10
test\spam10.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam11
test\spam11.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam12
test\spam12.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam13
test\spam13.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam14
test\spam14.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam15
test\spam15.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam16
test\spam16.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam17
test\spam17.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam18
test\spam18.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam19
test\spam19.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam2
test\spam2.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam20
test\spam20.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam3
test\spam3.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam4
test\spam4.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam5
test\spam5.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam6
test\spam6.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam7
test\spam7.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam8
test\spam8.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam9
test\spam9.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)

7. go to the parent directory of "test" and call "pylint -j 0 -E test":

(pylint) C:\work\pylint_test>pylint -j 0 -E test
************* Module test.spam11
test\spam11.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam14
test\spam14.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam12
test\spam12.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam11.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam17
test\spam17.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam10
test\spam10.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam1
test\spam1.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam14.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam18
test\spam18.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam13
test\spam13.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam16
test\spam16.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam15
test\spam15.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam11.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam17.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam2
test\spam2.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam12.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam19
test\spam19.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam10.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam20
test\spam20.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam14.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam18.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam4
test\spam4.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam1.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam3
test\spam3.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam13.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam5
test\spam5.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam15.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam7
test\spam7.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam11.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam17.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam2.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam8
test\spam8.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam12.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam19.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam9
test\spam9.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
test\spam16.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)
************* Module test.spam6
test\spam6.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable)

For example test\spam11.py:1:0: E0602: Undefined variable 'fdd' (undefined-variable) is reported four times.

Current behavior

The same error is reported multiple times.

Expected behavior

Each error is reported only once (and in the report at the correct place).

pylint --version output

pylint 2.6.0-dev1
astroid 2.4.0
Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40) [MSC v.1900 64 bit (AMD64)]

[NeilGirdhar]

For now, a simple workaround is to pipe the output of pylint through sort -u.

[cz-kialo]

We're seeing extreme numbers with ~14000 lines output on 6 CPUs of which only 132 are unique lines. Also the run times have increased very much from minutes to over an hour with 6 CPUs (probably related?).

The extreme behavior (including the running time multiplication) can be reproduced by extending the example above to higher numbers:
for i in {1..1000}; do cp spam.py "spam_$i.py"; done increases the error count from 1000 (--jobs 1) to

• 128000 (--jobs 2)
• 64000 (--jobs 4)
• 32000 (--jobs 8)

and the run time accordingly.
The fastest way to get the results at the moment is to run with --jobs 1. I suggest to rename the issue to "Using multiprocessing, e.g. --jobs 2 decreases performance dramatically".

pylint 2.5.2
astroid 2.4.1
Python 3.8.2 (default, Feb 28 2020, 14:51:45)

[ghost]

For now, a simple workaround is to pipe the output of pylint through sort -u.

What if we request stats?

[scop]

Any idea what's required to fix this, and when it might be addressed, or how could people help out with it?

The proposed workaround of piping through sort -u has the significant drawback of masking the exit status pretty much always to 0, thus possibly causing e.g. CI passes which shouldn't have been ones. Something like bash's pipefail option can be used to fix that, but that means a dependency on bash.