pylint-django calls ast_from_module_name without catching AstroidImportError

[jonholdsworth]

Bug description

Possibly related to: pylint-dev/pylint#8554

Pylint consistently crashes across all versions including latest, on one particular file, throwing 'AstroidError' mentioning 'inference_tip.py' and '_inference_tip_cached'.

Configuration

No response

Command used

pylint <directory>

Pylint output

I'm finding out if I'm allowed to post the actual code.

The error is:

pylint crashed with a ``AstroidError`` and with the following stacktrace:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/astroid/inference_tip.py", line 33, in _inference_tip_cached
    result = _cache[func, node]
KeyError: (<function infer_key_classes at 0x7ff9268d2c20>, <Call l.20 at 0x7ff921fd10f0>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pylint/checkers/utils.py", line 1371, in safe_infer
    value = next(infer_gen)
  File "/usr/local/lib/python3.10/site-packages/astroid/nodes/node_ng.py", line 171, in infer
    yield from self._infer(context=context, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/astroid/decorators.py", line 143, in raise_if_nothing_inferred
    yield next(generator)
  File "/usr/local/lib/python3.10/site-packages/astroid/decorators.py", line 112, in wrapped
    for res in _func(node, context, **kwargs):
  File "/usr/local/lib/python3.10/site-packages/astroid/bases.py", line 177, in _infer_stmts
    for inf in stmt.infer(context=context):
  File "/usr/local/lib/python3.10/site-packages/astroid/nodes/node_ng.py", line 161, in infer
    results = list(self._explicit_inference(self, context, **kwargs))
  File "/usr/local/lib/python3.10/site-packages/astroid/inference_tip.py", line 40, in _inference_tip_cached
    result = _cache[func, node] = list(func(*args, **kwargs))
  File "/usr/local/lib/python3.10/site-packages/pylint_django/transforms/foreignkey.py", line 120, in infer_key_classes
    MANAGER.ast_from_module_name(module_name)
  File "/usr/local/lib/python3.10/site-packages/astroid/manager.py", line 239, in ast_from_module_name
    raise e
  File "/usr/local/lib/python3.10/site-packages/astroid/manager.py", line 184, in ast_from_module_name
    found_spec = self.file_from_module_name(modname, context_file)
  File "/usr/local/lib/python3.10/site-packages/astroid/manager.py", line 292, in file_from_module_name
    raise value.with_traceback(None)  # pylint: disable=no-member
astroid.exceptions.AstroidImportError: Failed to import module projects.models with error:
No module named projects.models.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pylint/lint/pylinter.py", line 811, in _lint_file
    check_astroid_module(module)
  File "/usr/local/lib/python3.10/site-packages/pylint/lint/pylinter.py", line 1085, in check_astroid_module
    retval = self._check_astroid_module(
  File "/usr/local/lib/python3.10/site-packages/pylint/lint/pylinter.py", line 1135, in _check_astroid_module
    walker.walk(node)
  File "/usr/local/lib/python3.10/site-packages/pylint/utils/ast_walker.py", line 94, in walk
    self.walk(child)
  File "/usr/local/lib/python3.10/site-packages/pylint/utils/ast_walker.py", line 91, in walk
    callback(astroid)
  File "/usr/local/lib/python3.10/site-packages/pylint_plugin_utils/__init__.py", line 58, in __call__
    self.augmentation_func(Chain(self.old_method, node), node)
  File "/usr/local/lib/python3.10/site-packages/pylint_plugin_utils/__init__.py", line 115, in __call__
    chain()
  File "/usr/local/lib/python3.10/site-packages/pylint_plugin_utils/__init__.py", line 67, in __call__
    self.old_method(self.node)
  File "/usr/local/lib/python3.10/site-packages/pylint_plugin_utils/__init__.py", line 58, in __call__
    self.augmentation_func(Chain(self.old_method, node), node)
  File "/usr/local/lib/python3.10/site-packages/pylint_plugin_utils/__init__.py", line 115, in __call__
    chain()
  File "/usr/local/lib/python3.10/site-packages/pylint_plugin_utils/__init__.py", line 67, in __call__
    self.old_method(self.node)
  File "/usr/local/lib/python3.10/site-packages/pylint_plugin_utils/__init__.py", line 58, in __call__
    self.augmentation_func(Chain(self.old_method, node), node)
  File "/usr/local/lib/python3.10/site-packages/pylint_plugin_utils/__init__.py", line 115, in __call__
    chain()
  File "/usr/local/lib/python3.10/site-packages/pylint_plugin_utils/__init__.py", line 67, in __call__
    self.old_method(self.node)
  File "/usr/local/lib/python3.10/site-packages/pylint_plugin_utils/__init__.py", line 58, in __call__
    self.augmentation_func(Chain(self.old_method, node), node)
  File "/usr/local/lib/python3.10/site-packages/pylint_plugin_utils/__init__.py", line 115, in __call__
    chain()
  File "/usr/local/lib/python3.10/site-packages/pylint_plugin_utils/__init__.py", line 67, in __call__
    self.old_method(self.node)
  File "/usr/local/lib/python3.10/site-packages/pylint/checkers/classes/class_checker.py", line 876, in visit_classdef
    self._check_bases_classes(node)
  File "/usr/local/lib/python3.10/site-packages/pylint/checkers/classes/class_checker.py", line 2096, in _check_bases_classes
    unimplemented_abstract_methods(node, is_abstract).items(),
  File "/usr/local/lib/python3.10/site-packages/pylint/checkers/utils.py", line 972, in unimplemented_abstract_methods
    inferred = safe_infer(obj)
  File "/usr/local/lib/python3.10/site-packages/pylint/checkers/utils.py", line 1375, in safe_infer
    raise AstroidError from e
astroid.exceptions.AstroidError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pylint/lint/pylinter.py", line 775, in _lint_files
    self._lint_file(fileitem, module, check_astroid_module)
  File "/usr/local/lib/python3.10/site-packages/pylint/lint/pylinter.py", line 813, in _lint_file
    raise astroid.AstroidError from e
astroid.exceptions.AstroidError

Expected behavior

Pylint would just warn about code findings and not crash.

Pylint version

pylint 2.17.4 (got same result in 2.17.5 which I now don't seem to be able to install)
astroid 2.15.5
Python 3.10.11 (main, May 23 2023, 13:58:30) [GCC 10.2.1 20210110]

OS / Environment

Docker
Debian GNU/Linux 11

Additional dependencies

No response

[jonholdsworth]

We managed to fix this via config: with the following .pylintrc the error went away.
(There is no propriety information in this config so we are happy to share it.)
It looks like the problem was specific to our setup.
Please feel free to close this ticket if you wish, I will not complain.

[MASTER]
load-plugins=pylint_django
ignore=features, migrations
django-settings-module=config.settings

[REPORTS]
output-format=colorized

[FORMAT]
max-line-length=120

[MESSAGES CONTROL]
disable=
	RP0101,
	RP0204,
	RP0401,
	RP0402,
	RP0701,
	RP0801,
	W0707,
	C0209,
	R0022,
	R1735,
	W0640,
	R1721,
	W0236,
	W0719,
	C0415,
	R1729,
	R1732,
	W0236,
	R1724,
	R1720,
	W1514,
	R1720,
    attribute-defined-outside-init,
    fixme,
    invalid-name,
    locally-disabled,
    missing-docstring,
    similarities,
    too-few-public-methods,
    too-many-ancestors,
    too-many-lines,
    no-member,
    arguments-differ,
    unused-argument,
    too-many-arguments,
    too-many-instance-attributes,
    line-too-long,
    unnecessary-pass,
    cyclic-import,

[TYPECHECK]
generated-members=_meta,object,objects.request,kwargs,[a-z]+_set,DoesNotExist

[DESIGN]
max-parents=13

[DanielNoord]

@jacobtylerwalls Is this related to our latest changes to that cache?

[jacobtylerwalls]

I don't know yet. I'm planning to take a look this week.

[jacobtylerwalls]

Not related to the cache. (I should do a little refactor to avoid the KeyError from a cache miss showing up in the tracebacks like this.) EDIT: done, see pylint-dev/astroid#2205

It's just that safe_infer() doesn't catch AstroidImportError. (Or, that pylint-django doesn't catch it in their transform.)

[jacobtylerwalls]

pylint-django should catch and deal with AstroidImportError when calling ast_from_module_name. Should we also guard against it safe_infer()? I think it's better to guard against errors you reasonably expect, so probably not.

[carlio]

@jacobtylerwalls Hi there, I'm the one behind pylint-django, I'll look into this.

Do you know what specifically changed in your .pylintrc which made the problem go away?

[jacobtylerwalls]

Sorry, I'm not the reporter! But from this:

astroid.exceptions.AstroidImportError: Failed to import module projects.models with error:
No module named projects.models.

it seems like the change in the rc file made projects.models discoverable somehow.

[carlio]

Yes sorry, I picked the wrong autocomplete after typing @j :-)

@jonholdsworth Do you know what specifically changed in your .pylintrc which made the problem go away?

[jonholdsworth]

My apologies, I should have posted the diff in the first place.
It's from bitbucket so I'll have to paste this manually.

Hope this helps!

Changes were:

[MASTER]
load-plugins=pylint_django
ignore=features, migrations
+ django-settings-module=config.settings
[REPORTS]
output-format=colorized
[FORMAT]
max-line-length=120
[MESSAGES CONTROL]
disable=
	RP0101,
+ 	RP0204,
	RP0401,
	RP0402,
	RP0701,
	RP0801,
+ 	W0707,
+ 	C0209,
+ 	R0022,
+ 	R1735,
+ 	W0640,
+ 	R1721,
+ 	W0236,
+ 	W0719,
+ 	C0415,
+ 	R1729,
+ 	R1732,
+ 	W0236,
+ 	R1724,
+ 	R1720,
+ 	W1514,
+ 	R1720,
    attribute-defined-outside-init,
    fixme,
    invalid-name,
    locally-disabled,
-    locally-enabled,
    missing-docstring,
-     no-self-use,
-     redefined-variable-type,
    similarities,
    too-few-public-methods,
    too-many-ancestors,
    too-many-lines,
    no-member,
    arguments-differ,
-     bad-continuation,
    unused-argument,
    too-many-arguments,
    too-many-instance-attributes,
    line-too-long,
    unnecessary-pass,
    cyclic-import,
[TYPECHECK]
generated-members=_meta,object,objects.request,kwargs,[a-z]+_set,DoesNotExist
[DESIGN]
max-parents=13