Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Swap plugin cache to pickle-able values when done #7640

Merged
merged 9 commits into from
Oct 20, 2022
7 changes: 7 additions & 0 deletions doc/whatsnew/fragments/7635.bugfix
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
Fixes a multi-processing crash that prevents using any more than 1 thread on MacOS.
daogilvie marked this conversation as resolved.
Show resolved Hide resolved

The returned module objects and errors that were cached by the linter plugin loader
cannot be reliably pickled. This means that ``dill`` would throw an error when
attempting to serialise the linter object for multi-processing use.

Closes #7635.
14 changes: 12 additions & 2 deletions pylint/lint/pylinter.py
Original file line number Diff line number Diff line change
Expand Up @@ -296,7 +296,7 @@ def __init__(
str, list[checkers.BaseChecker]
] = collections.defaultdict(list)
"""Dictionary of registered and initialized checkers."""
self._dynamic_plugins: dict[str, ModuleType | ModuleNotFoundError] = {}
self._dynamic_plugins: dict[str, ModuleType | ModuleNotFoundError | bool] = {}
"""Set of loaded plugin names."""

# Attributes related to registering messages and their handling
Expand Down Expand Up @@ -402,7 +402,17 @@ def load_plugin_configuration(self) -> None:
"bad-plugin-value", args=(modname, module_or_error), line=0
)
elif hasattr(module_or_error, "load_configuration"):
module_or_error.load_configuration(self)
# Mypy is not smart enough to realize that we only call
# this line if the object has the attr we are looking at.
# Hence the one-line type: ignore[union-attr] here.
daogilvie marked this conversation as resolved.
Show resolved Hide resolved
cdce8p marked this conversation as resolved.
Show resolved Hide resolved
module_or_error.load_configuration(self) # type: ignore[union-attr]
# We re-set all the dictionary values to True here to make sure the dict
cdce8p marked this conversation as resolved.
Show resolved Hide resolved
# is pickle-able. This is only a problem in multiprocessing/parallel mode.
# (e.g. invoking pylint -j 2)
self._dynamic_plugins = {
modname: not isinstance(val, ModuleNotFoundError)
for modname, val in self._dynamic_plugins.items()
}

def _load_reporters(self, reporter_names: str) -> None:
"""Load the reporters if they are available on _reporters."""
Expand Down