Skip to content

Commit

Permalink
Make ModuleFinder try identifying non-PEP 561 packages
Browse files Browse the repository at this point in the history 
This pull request is an attempt at mitigating
python#4542 by making
mypy report a custom error when it detects installed
packages that are not PEP 561 compliant.

(I don't think it resolves it though -- I've come to
the conclusion that import handling is just inherently
complex/spooky. So if you were in a cynical mode, you
could perhaps argue the issue is just fundamentally
unresolvable...)

But anyways, this PR:

1.  Removes the hard-coded list of "popular third party
    libraries" from `moduleinfo.py` and replaces it with
    a heuristic that tries to find when an import "plausibly
    matches" some directory or Python file while we search
    for packages containing ``py.typed``.

    If we do find a plausible match, we generate an error
    that looks something like this:

    ```
    test.py:1: error: Skipping analyzing 'scipy': found module but no type hints or library stubs
    test.py:1: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
    ```

    The heuristic I'm using obviously isn't foolproof since
    we don't have any obvious signifiers like the ``py.typed``
    file we can look for, but it seemed to work well enough
    when I tried testing in on some of the libraries in the old list.

    Hopefully this will result in less confusion when users
    use a mix of "popular" and "unpopular" libraries.

2.  Gives us a way to add more fine-grained "module not found"
    error messages and heuristics in the future: we can add
    more entries to the ModuleNotFoundReason enum.

3.  Updates the docs about missing imports to use these new
    errors. I added a new subsection per each error type to
    try and make things a little less unwieldy.

4.  Adds what I think are common points of confusion to the
    doc -- e.g. that missing imports are inferred to be of
    type Any, what exactly it means to add a `# type: ignore`,
    and the whole virtualenv confusion thing.

5.  Modifies the docs to more strongly discourage the use
    of MYPYPATH. Unless I'm wrong, it's not a feature most
    people will find useful.

One limitation of this PR is that I added tests for just ModuleFinder.
I didn't want to dive into modifying our testcases framework to support
adding custom site-packages/some moral equivalent -- and my PR
only changes the behavior of ModuleFinder when it would have originally
reported something was not found, anyways.
  • Loading branch information
@Michael0x2a
Michael0x2a committed Jan 4, 2020
1 parent cdd91ba commit a4bf0db
Show file tree
Hide file tree
Showing 29 changed files with 405 additions and 336 deletions.
187 changes: 135 additions & 52 deletions docs/source/running_mypy.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -126,72 +126,102 @@ sections will discuss what to do in the other two cases.
.. _ignore-missing-imports:

Missing imports
---------------
***************

When you import a module, mypy may report that it is unable to
follow the import.

This can cause a lot of errors that look like the following::
This can cause errors that look like the following::

main.py:1: error: No library stub file for standard library module 'antigravity'
main.py:2: error: No library stub file for module 'flask'
main.py:2: error: Skipping analyzing 'django': found module but no type hints or library stubs
main.py:3: error: Cannot find implementation or library stub for module named 'this_module_does_not_exist'

There are several different things you can try doing, depending on the exact
nature of the module.
If you get any of these errors on an import, mypy will assume the type of that
module is ``Any``, the dynamic type. This means attempting to access any
attribute of the module will automatically succeed:

If the module is a part of your own codebase, try:
.. code-block:: python
1. Making sure your import does not contain a typo.
2. Reading the :ref:`finding-imports` section below to make sure you
understand how exactly mypy searches for and finds modules and modify
how you're invoking mypy accordingly.
3. Adding the directory containing that module to either the ``MYPYPATH``
environment variable or the ``mypy_path``
:ref:`config file option <config-file-import-discovery>`.
# Error: Cannot find implementation or library stub for module named 'does_not_exist'
import does_not_exist
Note: if the module you are trying to import is actually a *submodule* of
some package, you should add the directory containing the *entire* package
to ``MYPYPATH``. For example, suppose you are trying to add the module
``foo.bar.baz``, which is located at ``~/foo-project/src/foo/bar/baz.py``.
In this case, you should add ``~/foo-project/src`` to ``MYPYPATH``.

If the module is a third party library, you must make sure that there are
type hints available for that library. Mypy by default will not attempt to
infer the types of any 3rd party libraries you may have installed
# But this type checks, and x will have type 'Any'
x = does_not_exist.foobar()
The next three sections describe what each error means and recommended next steps.

Missing type hints for standard library module
----------------------------------------------

If you are getting a "No library stub file for standard library module" error,
this means that you are attempting to import something from the standard library
which has not yet been annotated with type hints. In this case, try:

1. Updating mypy and re-running it. It's possible type hints for that corner
of the standard library were added in a newer version of mypy.

2. Filing a bug report or submitting a pull request to
`typeshed <https://github.com/python/typeshed>`_, the repository of type hints
for the standard library that comes bundled with mypy.

Changes to typeshed will come bundled with mypy the next time it's released.
In the meantime, you can add a ``# type: ignore`` to the import to suppress
the errors generated on that line. After upgrading, run mypy with the
:option:`--warn-unused-ignores <mypy --warn-unused-ignores>` flag to help you
find any ``# type: ignore`` annotations you no longer need.

.. _missing-type-hints-for-third-party-library:

Missing type hints for third party library
------------------------------------------

If you are getting a "Skipping analyzing X: found module but no type hints or library stubs",
error, this means mypy was able to find the module you were importing, but no
corresponding type hints.

Mypy will not try inferring the types of any 3rd party libraries you have installed
unless they either have declared themselves to be
:ref:`PEP 561 compliant stub package <installed-packages>` or have registered
themselves on `typeshed <https://github.com/python/typeshed>`_,
the repository of types for the standard library and some 3rd party libraries.
themselves on `typeshed <https://github.com/python/typeshed>`_, the repository
of types for the standard library and some 3rd party libraries.

If you are getting an import-related error, this means the library you
are trying to use has done neither of these things. In that case, you can try:
If you are getting this error, try:

1. Searching to see if there is a :ref:`PEP 561 compliant stub package <installed-packages>`.
1. Upgrading the version of the library you're using, in case a newer version
has started to include type hints.

2. Searching to see if there is a :ref:`PEP 561 compliant stub package <installed-packages>`.
corresponding to your third party library. Stub packages let you install
type hints independently from the library itself.

2. :ref:`Writing your own stub files <stub-files>` containing type hints for
For example, if you want type hints for the ``django`` library, you can
install the `django-stubs <https://pypi.org/project/django-stubs/>`_ package.

3. :ref:`Writing your own stub files <stub-files>` containing type hints for
the library. You can point mypy at your type hints either by passing
them in via the command line, by adding the location to the
``MYPYPATH`` environment variable, or by using the ``mypy_path``
:ref:`config file option <config-file-import-discovery>`.
them in via the command line, by using the ``files`` or ``mypy_path``
:ref:`config file options <config-file-import-discovery>`, or by
adding the location to the ``MYPYPATH`` environment variable.

Note that if you decide to write your own stub files, they don't need
to be complete! A good strategy is to add stubs for just the parts
of the library you need and iterate on them over time.
These stub files do not need to be complete! A good strategy is to use
stubgen, a program that comes bundled with mypy, to generate a first
rough draft of the stubs. You can then iterate on just the parts of the
library you need.

If you want to share your work, you can try contributing your stubs back
to the library -- see our documentation on creating
:ref:`PEP 561 compliant packages <installed-packages>`.

If the module is a third party library, but you cannot find any existing
type hints nor have time to write your own, you can *silence* the errors:
If you are unable to find any existing type hints nor have time to write your
own, you can instead *suppress* the errors. All this will do is make mypy stop
reporting an error on the line containing the import: the imported module
will continue to be of type ``Any``.

1. To silence a *single* missing import error, add a ``# type: ignore`` at the end of the
1. To suppress a *single* missing import error, add a ``# type: ignore`` at the end of the
line containing the import.

2. To silence *all* missing import imports errors from a single library, add
2. To suppress *all* missing import imports errors from a single library, add
a section to your :ref:`mypy config file <config-file>` for that library setting
``ignore_missing_imports`` to True. For example, suppose your codebase
makes heavy use of an (untyped) library named ``foobar``. You can silence
Expand All @@ -206,7 +236,7 @@ type hints nor have time to write your own, you can *silence* the errors:
documentation about configuring
:ref:`import discovery <config-file-import-discovery>` in config files.

3. To silence *all* missing import errors for *all* libraries in your codebase,
3. To suppress *all* missing import errors for *all* libraries in your codebase,
invoke mypy with the :option:`--ignore-missing-imports <mypy --ignore-missing-imports>` command line flag or set
the ``ignore_missing_imports``
:ref:`config file option <config-file-import-discovery>` to True
Expand All @@ -218,26 +248,59 @@ type hints nor have time to write your own, you can *silence* the errors:
We recommend using this approach only as a last resort: it's equivalent
to adding a ``# type: ignore`` to all unresolved imports in your codebase.

If the module is a part of the standard library, try:
Unable to find module
---------------------

1. Updating mypy and re-running it. It's possible type hints for that corner
of the standard library were added in a later version of mypy.
If you are getting a "Cannot find implementation or library stub for module"
error, this means mypy was not able to find the module you are trying to
import, whether it comes bundled with type hints or not. If you are getting
this error, try:

2. Filing a bug report on `typeshed <https://github.com/python/typeshed>`_,
the repository of type hints for the standard library that comes bundled
with mypy. You can expedite this process by also submitting a pull request
fixing the bug.
1. Making sure your import does not contain a typo.

Changes to typeshed will come bundled with mypy the next time it's released.
In the meantime, you can add a ``# type: ignore`` to silence any relevant
errors. After upgrading, we recommend running mypy using the
:option:`--warn-unused-ignores <mypy --warn-unused-ignores>` flag to help you find any ``# type: ignore``
annotations you no longer need.
2. If the module is a third party library, making sure that mypy is able
to find the interpreter containing the installed library.

For example, if you are running your code in a virtualenv, make sure
to install and use mypy within the virtualenv. Alternatively, if you
want to use a globally installed mypy, set the
:option:`--python-executable <mypy --python-executable>` command
line flag to point the Python interpreter containing your installed
third party packages.

2. Reading the :ref:`finding-imports` section below to make sure you
understand how exactly mypy searches for and finds modules and modify
how you're invoking mypy accordingly.

3. Directly specifying the directory containing the module you want to
type check from the command line, by using the ``files`` or
``mypy_path`` :ref:`config file options <config-file-import-discovery>`,
or by using the ``MYPYPATH`` environment variable.

Note: if the module you are trying to import is actually a *submodule* of
some package, you should specific the directory containing the *entire* package.
For example, suppose you are trying to add the module ``foo.bar.baz``
which is located at ``~/foo-project/src/foo/bar/baz.py``. In this case,
you must run ``mypy ~/foo-project/src`` (or set the ``MYPYPATH`` to
``~/foo-project/src``.

4. If you are using namespace packages -- packages which do not contain
``__init__.py`` files within each subfolder -- using the
:option:`--namespace-package <mypy --namespace-package>` command
line flag.

In some rare cases, you may get the "Cannot find implementation or library
stub for module" error even when the module is installed in your system.
This can happen when the module is both missing type hints and is installed
on your system in a unconventional way.

In this case, follow the steps above on how to handle
:ref:`missing type hints in third party libraries <missing-type-hints-for-third-party-library>`.

.. _follow-imports:

Following imports
-----------------
*****************

Mypy is designed to :ref:`doggedly follow all imports <finding-imports>`,
even if the imported module is not a file you explicitly wanted mypy to check.
Expand Down Expand Up @@ -401,3 +464,23 @@ same directory on the search path, only the stub file is used.
(However, if the files are in different directories, the one found
in the earlier directory is used.)

Other advice and best practices
*******************************

There are multiple ways of telling mypy what files to type check, ranging
from passing in command line arguments to using the ``files`` or ``mypy_path``
:ref:`config file options <config-file-import-discovery>` to setting the
``MYPYPATH`` environment variable.

However, in practice, it is usually sufficient to just use either
command line arguments or the ``files`` config file option (the two
are largely interchangeable).

Setting ``mypy_path``/``MYPYPATH`` is mostly useful in the case
where you want to try running mypy against multiple distinct
sets of files that happen to share some common dependencies.

For example, if you have multiple projects that happen to be
using the same set of work-in-progress stubs, it could be
convenient to just have your ``MYPYPATH`` point to a single
directory containing the stubs.
62 changes: 30 additions & 32 deletions mypy/build.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,10 @@
from mypy.report import Reports # Avoid unconditional slow import
from mypy import moduleinfo
from mypy.fixup import fixup_module
from mypy.modulefinder import BuildSource, compute_search_paths, FindModuleCache, SearchPaths
from mypy.modulefinder import (
BuildSource, compute_search_paths, FindModuleCache, SearchPaths, ModuleSearchResult,
ModuleNotFoundReason
)
from mypy.nodes import Expression
from mypy.options import Options
from mypy.parse import parse
Expand Down Expand Up @@ -2337,40 +2340,40 @@ def find_module_and_diagnose(manager: BuildManager,
# difference and just assume 'builtins' everywhere,
# which simplifies code.
file_id = '__builtin__'
path = find_module_simple(file_id, manager)
if path:
result = find_module_with_reason(file_id, manager)
if isinstance(result, str):
# For non-stubs, look at options.follow_imports:
# - normal (default) -> fully analyze
# - silent -> analyze but silence errors
# - skip -> don't analyze, make the type Any
follow_imports = options.follow_imports
if (root_source # Honor top-level modules
or (not path.endswith('.py') # Stubs are always normal
or (not result.endswith('.py') # Stubs are always normal
and not options.follow_imports_for_stubs) # except when they aren't
or id in mypy.semanal_main.core_modules): # core is always normal
follow_imports = 'normal'
if skip_diagnose:
pass
elif follow_imports == 'silent':
# Still import it, but silence non-blocker errors.
manager.log("Silencing %s (%s)" % (path, id))
manager.log("Silencing %s (%s)" % (result, id))
elif follow_imports == 'skip' or follow_imports == 'error':
# In 'error' mode, produce special error messages.
if id not in manager.missing_modules:
manager.log("Skipping %s (%s)" % (path, id))
manager.log("Skipping %s (%s)" % (result, id))
if follow_imports == 'error':
if ancestor_for:
skipping_ancestor(manager, id, path, ancestor_for)
skipping_ancestor(manager, id, result, ancestor_for)
else:
skipping_module(manager, caller_line, caller_state,
id, path)
id, result)
raise ModuleNotFound
if not manager.options.no_silence_site_packages:
for dir in manager.search_paths.package_path + manager.search_paths.typeshed_path:
if is_sub_path(path, dir):
if is_sub_path(result, dir):
# Silence errors in site-package dirs and typeshed
follow_imports = 'silent'
return (path, follow_imports)
return (result, follow_imports)
else:
# Could not find a module. Typically the reason is a
# misspelled module name, missing stub, module not in
Expand All @@ -2379,7 +2382,7 @@ def find_module_and_diagnose(manager: BuildManager,
raise ModuleNotFound
if caller_state:
if not (options.ignore_missing_imports or in_partial_package(id, manager)):
module_not_found(manager, caller_line, caller_state, id)
module_not_found(manager, caller_line, caller_state, id, result)
raise ModuleNotFound
elif root_source:
# If we can't find a root source it's always fatal.
Expand Down Expand Up @@ -2416,10 +2419,17 @@ def exist_added_packages(suppressed: List[str],

def find_module_simple(id: str, manager: BuildManager) -> Optional[str]:
"""Find a filesystem path for module `id` or `None` if not found."""
x = find_module_with_reason(id, manager)
if isinstance(x, ModuleNotFoundReason):
return None
return x


def find_module_with_reason(id: str, manager: BuildManager) -> ModuleSearchResult:
"""Find a filesystem path for module `id` or the reason it can't be found."""
t0 = time.time()
x = manager.find_module_cache.find_module(id)
manager.add_stats(find_module_time=time.time() - t0, find_module_calls=1)

return x


Expand Down Expand Up @@ -2453,35 +2463,23 @@ def in_partial_package(id: str, manager: BuildManager) -> bool:


def module_not_found(manager: BuildManager, line: int, caller_state: State,
target: str) -> None:
target: str, reason: ModuleNotFoundReason) -> None:
errors = manager.errors
save_import_context = errors.import_context()
errors.set_import_context(caller_state.import_context)
errors.set_file(caller_state.xpath, caller_state.id)
stub_msg = "(Stub files are from https://github.com/python/typeshed)"
if target == 'builtins':
errors.report(line, 0, "Cannot find 'builtins' module. Typeshed appears broken!",
blocker=True)
errors.raise_error()
elif ((manager.options.python_version[0] == 2 and moduleinfo.is_py2_std_lib_module(target))
or (manager.options.python_version[0] >= 3
and moduleinfo.is_py3_std_lib_module(target))):
errors.report(
line, 0, "No library stub file for standard library module '{}'".format(target),
code=codes.IMPORT)
errors.report(line, 0, stub_msg, severity='note', only_once=True, code=codes.IMPORT)
elif moduleinfo.is_third_party_module(target):
errors.report(line, 0, "No library stub file for module '{}'".format(target),
code=codes.IMPORT)
errors.report(line, 0, stub_msg, severity='note', only_once=True, code=codes.IMPORT)
elif moduleinfo.is_std_lib_module(manager.options.python_version, target):
msg = "No library stub file for standard library module '{}'".format(target)
note = "(Stub files are from https://github.com/python/typeshed)"
errors.report(line, 0, msg, code=codes.IMPORT)
errors.report(line, 0, note, severity='note', only_once=True, code=codes.IMPORT)
else:
note = "See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports"
errors.report(
line,
0,
"Cannot find implementation or library stub for module named '{}'".format(target),
code=codes.IMPORT
)
msg, note = reason.error_message_templates()
errors.report(line, 0, msg.format(target), code=codes.IMPORT)
errors.report(line, 0, note, severity='note', only_once=True, code=codes.IMPORT)
errors.set_import_context(save_import_context)

Expand Down
Loading

0 comments on commit a4bf0db

Please sign in to comment.