diff --git a/docs/source/running_mypy.rst b/docs/source/running_mypy.rst index 64e7d39c0d72a..33133d42d997c 100644 --- a/docs/source/running_mypy.rst +++ b/docs/source/running_mypy.rst @@ -126,72 +126,102 @@ sections will discuss what to do in the other two cases. .. _ignore-missing-imports: Missing imports ---------------- +*************** When you import a module, mypy may report that it is unable to follow the import. -This can cause a lot of errors that look like the following:: +This can cause errors that look like the following:: main.py:1: error: No library stub file for standard library module 'antigravity' - main.py:2: error: No library stub file for module 'flask' + main.py:2: error: Skipping analyzing 'django': found module but no type hints or library stubs main.py:3: error: Cannot find implementation or library stub for module named 'this_module_does_not_exist' -There are several different things you can try doing, depending on the exact -nature of the module. +If you get any of these errors on an import, mypy will assume the type of that +module is ``Any``, the dynamic type. This means attempting to access any +attribute of the module will automatically succeed: -If the module is a part of your own codebase, try: +.. code-block:: python -1. Making sure your import does not contain a typo. -2. Reading the :ref:`finding-imports` section below to make sure you - understand how exactly mypy searches for and finds modules and modify - how you're invoking mypy accordingly. -3. Adding the directory containing that module to either the ``MYPYPATH`` - environment variable or the ``mypy_path`` - :ref:`config file option `. + # Error: Cannot find implementation or library stub for module named 'does_not_exist' + import does_not_exist - Note: if the module you are trying to import is actually a *submodule* of - some package, you should add the directory containing the *entire* package - to ``MYPYPATH``. For example, suppose you are trying to add the module - ``foo.bar.baz``, which is located at ``~/foo-project/src/foo/bar/baz.py``. - In this case, you should add ``~/foo-project/src`` to ``MYPYPATH``. - -If the module is a third party library, you must make sure that there are -type hints available for that library. Mypy by default will not attempt to -infer the types of any 3rd party libraries you may have installed + # But this type checks, and x will have type 'Any' + x = does_not_exist.foobar() + +The next three sections describe what each error means and recommended next steps. + +Missing type hints for standard library module +---------------------------------------------- + +If you are getting a "No library stub file for standard library module" error, +this means that you are attempting to import something from the standard library +which has not yet been annotated with type hints. In this case, try: + +1. Updating mypy and re-running it. It's possible type hints for that corner + of the standard library were added in a newer version of mypy. + +2. Filing a bug report or submitting a pull request to + `typeshed `_, the repository of type hints + for the standard library that comes bundled with mypy. + + Changes to typeshed will come bundled with mypy the next time it's released. + In the meantime, you can add a ``# type: ignore`` to the import to suppress + the errors generated on that line. After upgrading, run mypy with the + :option:`--warn-unused-ignores ` flag to help you + find any ``# type: ignore`` annotations you no longer need. + +.. _missing-type-hints-for-third-party-library: + +Missing type hints for third party library +------------------------------------------ + +If you are getting a "Skipping analyzing X: found module but no type hints or library stubs", +error, this means mypy was able to find the module you were importing, but no +corresponding type hints. + +Mypy will not try inferring the types of any 3rd party libraries you have installed unless they either have declared themselves to be :ref:`PEP 561 compliant stub package ` or have registered -themselves on `typeshed `_, -the repository of types for the standard library and some 3rd party libraries. +themselves on `typeshed `_, the repository +of types for the standard library and some 3rd party libraries. -If you are getting an import-related error, this means the library you -are trying to use has done neither of these things. In that case, you can try: +If you are getting this error, try: -1. Searching to see if there is a :ref:`PEP 561 compliant stub package `. +1. Upgrading the version of the library you're using, in case a newer version + has started to include type hints. + +2. Searching to see if there is a :ref:`PEP 561 compliant stub package `. corresponding to your third party library. Stub packages let you install type hints independently from the library itself. -2. :ref:`Writing your own stub files ` containing type hints for + For example, if you want type hints for the ``django`` library, you can + install the `django-stubs `_ package. + +3. :ref:`Writing your own stub files ` containing type hints for the library. You can point mypy at your type hints either by passing - them in via the command line, by adding the location to the - ``MYPYPATH`` environment variable, or by using the ``mypy_path`` - :ref:`config file option `. + them in via the command line, by using the ``files`` or ``mypy_path`` + :ref:`config file options `, or by + adding the location to the ``MYPYPATH`` environment variable. - Note that if you decide to write your own stub files, they don't need - to be complete! A good strategy is to add stubs for just the parts - of the library you need and iterate on them over time. + These stub files do not need to be complete! A good strategy is to use + stubgen, a program that comes bundled with mypy, to generate a first + rough draft of the stubs. You can then iterate on just the parts of the + library you need. If you want to share your work, you can try contributing your stubs back to the library -- see our documentation on creating :ref:`PEP 561 compliant packages `. -If the module is a third party library, but you cannot find any existing -type hints nor have time to write your own, you can *silence* the errors: +If you are unable to find any existing type hints nor have time to write your +own, you can instead *suppress* the errors. All this will do is make mypy stop +reporting an error on the line containing the import: the imported module +will continue to be of type ``Any``. -1. To silence a *single* missing import error, add a ``# type: ignore`` at the end of the +1. To suppress a *single* missing import error, add a ``# type: ignore`` at the end of the line containing the import. -2. To silence *all* missing import imports errors from a single library, add +2. To suppress *all* missing import imports errors from a single library, add a section to your :ref:`mypy config file ` for that library setting ``ignore_missing_imports`` to True. For example, suppose your codebase makes heavy use of an (untyped) library named ``foobar``. You can silence @@ -206,7 +236,7 @@ type hints nor have time to write your own, you can *silence* the errors: documentation about configuring :ref:`import discovery ` in config files. -3. To silence *all* missing import errors for *all* libraries in your codebase, +3. To suppress *all* missing import errors for *all* libraries in your codebase, invoke mypy with the :option:`--ignore-missing-imports ` command line flag or set the ``ignore_missing_imports`` :ref:`config file option ` to True @@ -218,26 +248,59 @@ type hints nor have time to write your own, you can *silence* the errors: We recommend using this approach only as a last resort: it's equivalent to adding a ``# type: ignore`` to all unresolved imports in your codebase. -If the module is a part of the standard library, try: +Unable to find module +--------------------- -1. Updating mypy and re-running it. It's possible type hints for that corner - of the standard library were added in a later version of mypy. +If you are getting a "Cannot find implementation or library stub for module" +error, this means mypy was not able to find the module you are trying to +import, whether it comes bundled with type hints or not. If you are getting +this error, try: -2. Filing a bug report on `typeshed `_, - the repository of type hints for the standard library that comes bundled - with mypy. You can expedite this process by also submitting a pull request - fixing the bug. +1. Making sure your import does not contain a typo. - Changes to typeshed will come bundled with mypy the next time it's released. - In the meantime, you can add a ``# type: ignore`` to silence any relevant - errors. After upgrading, we recommend running mypy using the - :option:`--warn-unused-ignores ` flag to help you find any ``# type: ignore`` - annotations you no longer need. +2. If the module is a third party library, making sure that mypy is able + to find the interpreter containing the installed library. + + For example, if you are running your code in a virtualenv, make sure + to install and use mypy within the virtualenv. Alternatively, if you + want to use a globally installed mypy, set the + :option:`--python-executable ` command + line flag to point the Python interpreter containing your installed + third party packages. + +2. Reading the :ref:`finding-imports` section below to make sure you + understand how exactly mypy searches for and finds modules and modify + how you're invoking mypy accordingly. + +3. Directly specifying the directory containing the module you want to + type check from the command line, by using the ``files`` or + ``mypy_path`` :ref:`config file options `, + or by using the ``MYPYPATH`` environment variable. + + Note: if the module you are trying to import is actually a *submodule* of + some package, you should specific the directory containing the *entire* package. + For example, suppose you are trying to add the module ``foo.bar.baz`` + which is located at ``~/foo-project/src/foo/bar/baz.py``. In this case, + you must run ``mypy ~/foo-project/src`` (or set the ``MYPYPATH`` to + ``~/foo-project/src``. + +4. If you are using namespace packages -- packages which do not contain + ``__init__.py`` files within each subfolder -- using the + :option:`--namespace-package ` command + line flag. + +In some rare cases, you may get the "Cannot find implementation or library +stub for module" error even when the module is installed in your system. +This can happen when the module is both missing type hints and is installed +on your system in a unconventional way. + +In this case, follow the steps above on how to handle +:ref:`missing type hints in third party libraries `. .. _follow-imports: Following imports ------------------ +***************** Mypy is designed to :ref:`doggedly follow all imports `, even if the imported module is not a file you explicitly wanted mypy to check. @@ -401,3 +464,23 @@ same directory on the search path, only the stub file is used. (However, if the files are in different directories, the one found in the earlier directory is used.) +Other advice and best practices +******************************* + +There are multiple ways of telling mypy what files to type check, ranging +from passing in command line arguments to using the ``files`` or ``mypy_path`` +:ref:`config file options ` to setting the +``MYPYPATH`` environment variable. + +However, in practice, it is usually sufficient to just use either +command line arguments or the ``files`` config file option (the two +are largely interchangeable). + +Setting ``mypy_path``/``MYPYPATH`` is mostly useful in the case +where you want to try running mypy against multiple distinct +sets of files that happen to share some common dependencies. + +For example, if you have multiple projects that happen to be +using the same set of work-in-progress stubs, it could be +convenient to just have your ``MYPYPATH`` point to a single +directory containing the stubs. diff --git a/mypy/build.py b/mypy/build.py index 749785907f02a..ba416340e6f57 100644 --- a/mypy/build.py +++ b/mypy/build.py @@ -43,7 +43,10 @@ from mypy.report import Reports # Avoid unconditional slow import from mypy import moduleinfo from mypy.fixup import fixup_module -from mypy.modulefinder import BuildSource, compute_search_paths, FindModuleCache, SearchPaths +from mypy.modulefinder import ( + BuildSource, compute_search_paths, FindModuleCache, SearchPaths, ModuleSearchResult, + ModuleNotFoundReason +) from mypy.nodes import Expression from mypy.options import Options from mypy.parse import parse @@ -2337,15 +2340,15 @@ def find_module_and_diagnose(manager: BuildManager, # difference and just assume 'builtins' everywhere, # which simplifies code. file_id = '__builtin__' - path = find_module_simple(file_id, manager) - if path: + result = find_module_with_reason(file_id, manager) + if isinstance(result, str): # For non-stubs, look at options.follow_imports: # - normal (default) -> fully analyze # - silent -> analyze but silence errors # - skip -> don't analyze, make the type Any follow_imports = options.follow_imports if (root_source # Honor top-level modules - or (not path.endswith('.py') # Stubs are always normal + or (not result.endswith('.py') # Stubs are always normal and not options.follow_imports_for_stubs) # except when they aren't or id in mypy.semanal_main.core_modules): # core is always normal follow_imports = 'normal' @@ -2353,24 +2356,24 @@ def find_module_and_diagnose(manager: BuildManager, pass elif follow_imports == 'silent': # Still import it, but silence non-blocker errors. - manager.log("Silencing %s (%s)" % (path, id)) + manager.log("Silencing %s (%s)" % (result, id)) elif follow_imports == 'skip' or follow_imports == 'error': # In 'error' mode, produce special error messages. if id not in manager.missing_modules: - manager.log("Skipping %s (%s)" % (path, id)) + manager.log("Skipping %s (%s)" % (result, id)) if follow_imports == 'error': if ancestor_for: - skipping_ancestor(manager, id, path, ancestor_for) + skipping_ancestor(manager, id, result, ancestor_for) else: skipping_module(manager, caller_line, caller_state, - id, path) + id, result) raise ModuleNotFound if not manager.options.no_silence_site_packages: for dir in manager.search_paths.package_path + manager.search_paths.typeshed_path: - if is_sub_path(path, dir): + if is_sub_path(result, dir): # Silence errors in site-package dirs and typeshed follow_imports = 'silent' - return (path, follow_imports) + return (result, follow_imports) else: # Could not find a module. Typically the reason is a # misspelled module name, missing stub, module not in @@ -2379,7 +2382,7 @@ def find_module_and_diagnose(manager: BuildManager, raise ModuleNotFound if caller_state: if not (options.ignore_missing_imports or in_partial_package(id, manager)): - module_not_found(manager, caller_line, caller_state, id) + module_not_found(manager, caller_line, caller_state, id, result) raise ModuleNotFound elif root_source: # If we can't find a root source it's always fatal. @@ -2416,10 +2419,17 @@ def exist_added_packages(suppressed: List[str], def find_module_simple(id: str, manager: BuildManager) -> Optional[str]: """Find a filesystem path for module `id` or `None` if not found.""" + x = find_module_with_reason(id, manager) + if isinstance(x, ModuleNotFoundReason): + return None + return x + + +def find_module_with_reason(id: str, manager: BuildManager) -> ModuleSearchResult: + """Find a filesystem path for module `id` or the reason it can't be found.""" t0 = time.time() x = manager.find_module_cache.find_module(id) manager.add_stats(find_module_time=time.time() - t0, find_module_calls=1) - return x @@ -2453,35 +2463,23 @@ def in_partial_package(id: str, manager: BuildManager) -> bool: def module_not_found(manager: BuildManager, line: int, caller_state: State, - target: str) -> None: + target: str, reason: ModuleNotFoundReason) -> None: errors = manager.errors save_import_context = errors.import_context() errors.set_import_context(caller_state.import_context) errors.set_file(caller_state.xpath, caller_state.id) - stub_msg = "(Stub files are from https://github.com/python/typeshed)" if target == 'builtins': errors.report(line, 0, "Cannot find 'builtins' module. Typeshed appears broken!", blocker=True) errors.raise_error() - elif ((manager.options.python_version[0] == 2 and moduleinfo.is_py2_std_lib_module(target)) - or (manager.options.python_version[0] >= 3 - and moduleinfo.is_py3_std_lib_module(target))): - errors.report( - line, 0, "No library stub file for standard library module '{}'".format(target), - code=codes.IMPORT) - errors.report(line, 0, stub_msg, severity='note', only_once=True, code=codes.IMPORT) - elif moduleinfo.is_third_party_module(target): - errors.report(line, 0, "No library stub file for module '{}'".format(target), - code=codes.IMPORT) - errors.report(line, 0, stub_msg, severity='note', only_once=True, code=codes.IMPORT) + elif moduleinfo.is_std_lib_module(manager.options.python_version, target): + msg = "No library stub file for standard library module '{}'".format(target) + note = "(Stub files are from https://github.com/python/typeshed)" + errors.report(line, 0, msg, code=codes.IMPORT) + errors.report(line, 0, note, severity='note', only_once=True, code=codes.IMPORT) else: - note = "See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports" - errors.report( - line, - 0, - "Cannot find implementation or library stub for module named '{}'".format(target), - code=codes.IMPORT - ) + msg, note = reason.error_message_templates() + errors.report(line, 0, msg.format(target), code=codes.IMPORT) errors.report(line, 0, note, severity='note', only_once=True, code=codes.IMPORT) errors.set_import_context(save_import_context) diff --git a/mypy/modulefinder.py b/mypy/modulefinder.py index 8802c2a2eb5e9..8b4a6f2715457 100644 --- a/mypy/modulefinder.py +++ b/mypy/modulefinder.py @@ -9,8 +9,9 @@ import os import subprocess import sys +from enum import Enum -from typing import Dict, List, NamedTuple, Optional, Set, Tuple +from typing import Dict, List, NamedTuple, Optional, Set, Tuple, Union from typing_extensions import Final from mypy.defaults import PYTHON3_VERSION_MIN @@ -34,6 +35,37 @@ PYTHON_EXTENSIONS = ['.pyi', '.py'] # type: Final +# TODO: Consider adding more reasons here? +# E.g. if we deduce a module would likely be found if the user were +# to set the --namespace-packages flag. +class ModuleNotFoundReason(Enum): + # The module was not found: we found neither stubs nor a plausible code + # implementation (with or without a py.typed file). + NOT_FOUND = 0 + + # The implementation for this module plausibly exists (e.g. we + # found a matching folder or *.py file), but either the parent package + # did not contain a py.typed file or we were unable to find a + # corresponding *-stubs package. + FOUND_WITHOUT_TYPE_HINTS = 1 + + def error_message_templates(self) -> Tuple[str, str]: + if self is ModuleNotFoundReason.NOT_FOUND: + msg = "Cannot find implementation or library stub for module named '{}'" + note = "See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports" + elif self is ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS: + msg = "Skipping analyzing '{}': found module but no type hints or library stubs" + note = "See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports" + else: + assert False + return msg, note + + +# If we found the module, returns the path to the module as a str. +# Otherwise, returns the reason why the module wasn't found. +ModuleSearchResult = Union[str, ModuleNotFoundReason] + + class BuildSource: """A single source file.""" @@ -72,7 +104,7 @@ def __init__(self, # search_paths -> (toplevel_id -> list(package_dirs)) self.initial_components = {} # type: Dict[Tuple[str, ...], Dict[str, List[str]]] # Cache find_module: id -> result - self.results = {} # type: Dict[str, Optional[str]] + self.results = {} # type: Dict[str, ModuleSearchResult] self.ns_ancestors = {} # type: Dict[str, str] self.options = options self.ns_packages = ns_packages or [] # type: List[str] @@ -128,20 +160,27 @@ def get_toplevel_possibilities(self, lib_path: Tuple[str, ...], id: str) -> List self.initial_components[lib_path] = components return components.get(id, []) - def find_module(self, id: str) -> Optional[str]: - """Return the path of the module source file, or None if not found.""" + def find_module(self, id: str) -> ModuleSearchResult: + """Return the path of the module source file or why it wasn't found.""" if id not in self.results: self.results[id] = self._find_module(id) return self.results[id] def _find_module_non_stub_helper(self, components: List[str], - pkg_dir: str) -> Optional[OnePackageDir]: + pkg_dir: str) -> Union[OnePackageDir, ModuleNotFoundReason]: + plausible_match = False dir_path = pkg_dir for index, component in enumerate(components): dir_path = os.path.join(dir_path, component) if self.fscache.isfile(os.path.join(dir_path, 'py.typed')): return os.path.join(pkg_dir, *components[:-1]), index == 0 - return None + elif not plausible_match and (self.fscache.isdir(dir_path) + or self.fscache.isfile(dir_path + ".py")): + plausible_match = True + if plausible_match: + return ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS + else: + return ModuleNotFoundReason.NOT_FOUND def _update_ns_ancestors(self, components: List[str], match: Tuple[str, bool]) -> None: path, verify = match @@ -151,7 +190,7 @@ def _update_ns_ancestors(self, components: List[str], match: Tuple[str, bool]) - self.ns_ancestors[pkg_id] = path path = os.path.dirname(path) - def _find_module(self, id: str) -> Optional[str]: + def _find_module(self, id: str) -> ModuleSearchResult: fscache = self.fscache # If we're looking for a module like 'foo.bar.baz', it's likely that most of the @@ -166,6 +205,7 @@ def _find_module(self, id: str) -> Optional[str]: # put them in the front of the search path third_party_inline_dirs = [] # type: PackageDirs third_party_stubs_dirs = [] # type: PackageDirs + found_possible_third_party_missing_type_hints = False # Third-party stub/typed packages for pkg_dir in self.search_paths.package_path: stub_name = components[0] + '-stubs' @@ -193,13 +233,17 @@ def _find_module(self, id: str) -> Optional[str]: else: third_party_stubs_dirs.append((path, True)) non_stub_match = self._find_module_non_stub_helper(components, pkg_dir) - if non_stub_match: + if isinstance(non_stub_match, ModuleNotFoundReason): + if non_stub_match is ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS: + found_possible_third_party_missing_type_hints = True + else: third_party_inline_dirs.append(non_stub_match) self._update_ns_ancestors(components, non_stub_match) if self.options and self.options.use_builtins_fixtures: # Everything should be in fixtures. third_party_inline_dirs.clear() third_party_stubs_dirs.clear() + found_possible_third_party_missing_type_hints = False python_mypy_path = self.search_paths.mypy_path + self.search_paths.python_path candidate_base_dirs = self.find_lib_path_dirs(id, python_mypy_path) + \ third_party_stubs_dirs + third_party_inline_dirs + \ @@ -279,11 +323,18 @@ def _find_module(self, id: str) -> Optional[str]: # installed package with a py.typed marker that is a # subpackage of a namespace package. We only fess up to these # if we would otherwise return "not found". - return self.ns_ancestors.get(id) + ancestor = self.ns_ancestors.get(id) + if ancestor is not None: + return ancestor + + if found_possible_third_party_missing_type_hints: + return ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS + else: + return ModuleNotFoundReason.NOT_FOUND def find_modules_recursive(self, module: str) -> List[BuildSource]: module_path = self.find_module(module) - if not module_path: + if isinstance(module_path, ModuleNotFoundReason): return [] result = [BuildSource(module_path, module, None)] if module_path.endswith(('__init__.py', '__init__.pyi')): diff --git a/mypy/moduleinfo.py b/mypy/moduleinfo.py index a710037788e92..9cf45784ff040 100644 --- a/mypy/moduleinfo.py +++ b/mypy/moduleinfo.py @@ -12,221 +12,9 @@ no stub for a module. """ -from typing import Set +from typing import Set, Tuple from typing_extensions import Final -third_party_modules = { - # From https://hugovk.github.io/top-pypi-packages/ - 'pip', - 'urllib3', - 'six', - 'botocore', - 'dateutil', - 's3transfer', - 'yaml', - 'requests', - 'pyasn1', - 'docutils', - 'jmespath', - 'certifi', - 'rsa', - 'setuptools', - 'idna', - 'awscli', - 'concurrent.futures', - 'colorama', - 'chardet', - 'wheel', - 'simplejson', - 'boto3', - 'pytz', - 'numpy', - 'markupsafe', - 'jinja2', - 'cffi', - 'cryptography', - 'google.protobuf', - 'cwlogs', - 'enum', - 'pycparser', - 'asn1crypto', - 'attr', - 'click', - 'ipaddress', - 'pytest', - 'future', - 'decorator', - 'pbr', - 'google.api', - 'pandas', - 'werkzeug', - 'pyparsing', - 'flask', - 'psutil', - 'itsdangerous', - 'google.cloud', - 'grpc', - 'cachetools', - 'virtualenv', - 'google.auth', - 'py', - 'pluggy', - 'scipy', - 'boto', - 'coverage', - 'mock', - 'OpenSSL', - 'sklearn', - 'jsonschema', - 'argparse', - 'more_itertools', - 'pygments', - 'psycopg2', - 'websocket', - 'PIL', - 'googleapiclient', - 'httplib2', - 'matplotlib', - 'oauth2client', - 'docopt', - 'tornado', - 'funcsigs', - 'lxml', - 'prompt_toolkit', - 'paramiko', - 'jwt', - 'IPython', - 'docker', - 'dockerpycreds', - 'oauthlib', - 'mccabe', - 'google.resumable_media', - 'sqlalchemy', - 'wrapt', - 'bcrypt', - 'ptyprocess', - 'requests_oauthlib', - 'multidict', - 'markdown', - 'pexpect', - 'atomicwrites', - 'uritemplate', - 'nacl', - 'pycodestyle', - 'elasticsearch', - 'absl', - 'aiohttp', - 'redis', - 'sklearn', - 'gevent', - 'pymysql', - 'wcwidth', - 'tqdm', - 'bs4', - 'functools32', - 'configparser', - 'gunicorn', - 'typing', - 'ujson', - 'pyflakes', - 'packaging', - 'lazy_object_proxy', - 'ipython_genutils', - 'toolz', - 'async_timeout', - 'traitlets', - 'kiwisolver', - 'pathlib2', - 'greenlet', - 'networkx', - 'cv2', - 'termcolor', - 'babel', - 'django', - 'pymemcache', - 'skimage', - 'pickleshare', - 'flake8', - 'cycler', - 'requests_toolbelt', - 'bleach', - 'scandir', - 'selenium', - 'dask', - 'websockets', - 'isort', - 'h5py', - 'tabulate', - 'tensorflow', - 'html5lib', - 'pylint', - 'tensorboard', - 'compose', - 'astroid', - 'trueskill', - 'webencodings', - 'defusedxml', - 'pykube', - 'pymongo', - 'retrying', - 'cached_property', - 'zope', - 'singledispatch', - 'tzlocal', - 'datadog', - 'zmq', - 'discord', - 'apache_beam', - 'subprocess32', - 'astor', - 'entrypoints', - 'gast', - 'nose', - 'smmap', - 'gitdb', - 'isodate', - 'pywt', - 'simplegeneric', - 'sortedcontainers', - 'psycopg2', - 'pytest_cov', - 'hiredis', - 'elasticsearch_dsl', - 'dill', - 'keras', - 'contextlib2', - 'hdfs', - 'jupyter_core', - 'typed_ast', - 'croniter', - 'azure', - 'nbformat', - 'xmltodict', - 'lockfile', - 'arrow', - 'parso', - 'jsonpickle', - - # Skipped (name considered too generic): - # - fixtures - # - migrate (from sqlalchemy-migrate) - # - git (GitPython) - - # Other - 'formencode', - 'pkg_resources', - 'wx', - 'gi.repository', - 'pygtk', - 'gtk', - 'PyQt4', - 'PyQt5', - 'pylons', - - # for use in tests - '__dummy_third_party1', -} # type: Final - # Modules and packages common to Python 2.7 and 3.x. common_std_lib_modules = { 'abc', @@ -547,12 +335,14 @@ } # type: Final -def is_third_party_module(id: str) -> bool: - return is_in_module_collection(third_party_modules, id) - - -def is_py2_std_lib_module(id: str) -> bool: - return is_in_module_collection(python2_std_lib_modules, id) +def is_std_lib_module(python_version: Tuple[int, int], id: str) -> bool: + if python_version[0] == 2: + return is_in_module_collection(python2_std_lib_modules, id) + elif python_version[0] >= 3: + return is_in_module_collection(python3_std_lib_modules, id) + else: + # TODO: Raise an exception here? + return False def is_py3_std_lib_module(id: str) -> bool: diff --git a/mypy/stubgen.py b/mypy/stubgen.py index b86da770b4d3a..75fa94e8e6309 100755 --- a/mypy/stubgen.py +++ b/mypy/stubgen.py @@ -65,7 +65,9 @@ import mypy.mixedtraverser import mypy.util from mypy import defaults -from mypy.modulefinder import FindModuleCache, SearchPaths, BuildSource, default_lib_path +from mypy.modulefinder import ( + ModuleNotFoundReason, FindModuleCache, SearchPaths, BuildSource, default_lib_path +) from mypy.nodes import ( Expression, IntExpr, UnaryExpr, StrExpr, BytesExpr, NameExpr, FloatExpr, MemberExpr, TupleExpr, ListExpr, ComparisonExpr, CallExpr, IndexExpr, EllipsisExpr, @@ -1290,14 +1292,17 @@ def find_module_paths_using_search(modules: List[str], packages: List[str], search_paths = SearchPaths(('.',) + tuple(search_path), (), (), tuple(typeshed_path)) cache = FindModuleCache(search_paths) for module in modules: - module_path = cache.find_module(module) - if not module_path: - fail_missing(module) + m_result = cache.find_module(module) + if isinstance(m_result, ModuleNotFoundReason): + fail_missing(module, m_result) + module_path = None + else: + module_path = m_result result.append(StubSource(module, module_path)) for package in packages: p_result = cache.find_modules_recursive(package) - if not p_result: - fail_missing(package) + if p_result: + fail_missing(package, ModuleNotFoundReason.NOT_FOUND) sources = [StubSource(m.module, m.path) for m in p_result] result.extend(sources) diff --git a/mypy/stubutil.py b/mypy/stubutil.py index bf5de6d607e27..51f9ef6e39ff6 100644 --- a/mypy/stubutil.py +++ b/mypy/stubutil.py @@ -11,6 +11,7 @@ from typing_extensions import overload from mypy.moduleinspect import ModuleInspect, InspectError +from mypy.modulefinder import ModuleNotFoundReason # Modules that may fail when imported, or that may have side effects (fully qualified). @@ -195,8 +196,14 @@ def report_missing(mod: str, message: Optional[str] = '', traceback: str = '') - print('note: Try --py2 for Python 2 mode') -def fail_missing(mod: str) -> None: - raise SystemExit("Can't find module '{}' (consider using --search-path)".format(mod)) +def fail_missing(mod: str, reason: ModuleNotFoundReason) -> None: + if reason is ModuleNotFoundReason.NOT_FOUND: + clarification = "(consider using --search-path)" + elif reason is ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS: + clarification = "(module likely exists, but is not PEP 561 compatible)" + else: + clarification = "(unknown reason '{}')".format(reason) + raise SystemExit("Can't find module '{}' {}".format(mod, clarification)) @overload diff --git a/mypy/test/testcheck.py b/mypy/test/testcheck.py index 2747d1c034d1c..f969fb338c1ba 100644 --- a/mypy/test/testcheck.py +++ b/mypy/test/testcheck.py @@ -335,7 +335,7 @@ def parse_module(self, cache = FindModuleCache(search_paths) for module_name in module_names.split(' '): path = cache.find_module(module_name) - assert path is not None, "Can't find ad hoc case file" + assert isinstance(path, str), "Can't find ad hoc case file" with open(path, encoding='utf8') as f: program_text = f.read() out.append((module_name, path, program_text)) diff --git a/mypy/test/testmodulefinder.py b/mypy/test/testmodulefinder.py index ab1a0d8e67f4a..07d69891707ae 100644 --- a/mypy/test/testmodulefinder.py +++ b/mypy/test/testmodulefinder.py @@ -1,7 +1,7 @@ import os from mypy.options import Options -from mypy.modulefinder import FindModuleCache, SearchPaths +from mypy.modulefinder import FindModuleCache, SearchPaths, ModuleNotFoundReason from mypy.test.helpers import Suite, assert_equal from mypy.test.config import package_path @@ -38,14 +38,14 @@ def test__no_namespace_packages__nsx(self) -> None: If namespace_packages is False, we shouldn't find nsx """ found_module = self.fmc_nons.find_module("nsx") - self.assertIsNone(found_module) + assert_equal(ModuleNotFoundReason.NOT_FOUND, found_module) def test__no_namespace_packages__nsx_a(self) -> None: """ If namespace_packages is False, we shouldn't find nsx.a. """ found_module = self.fmc_nons.find_module("nsx.a") - self.assertIsNone(found_module) + assert_equal(ModuleNotFoundReason.NOT_FOUND, found_module) def test__no_namespace_packages__find_a_in_pkg1(self) -> None: """ @@ -133,4 +133,131 @@ def test__find_b_init_in_pkg2(self) -> None: def test__find_d_nowhere(self) -> None: found_module = self.fmc_ns.find_module("d") - self.assertIsNone(found_module) + assert_equal(ModuleNotFoundReason.NOT_FOUND, found_module) + + +class ModuleFinderSitePackagesSuite(Suite): + + def setUp(self) -> None: + self.package_dir = os.path.relpath(os.path.join( + package_path, + "modulefinder-site-packages", + )) + self.search_paths = SearchPaths( + python_path=(), + mypy_path=( + os.path.join(data_path, "pkg1"), + ), + package_path=( + self.package_dir, + ), + typeshed_path=(), + ) + options = Options() + options.namespace_packages = True + self.fmc_ns = FindModuleCache(self.search_paths, options=options) + + options = Options() + options.namespace_packages = False + self.fmc_nons = FindModuleCache(self.search_paths, options=options) + + def path(self, suffix: str) -> str: + return os.path.join(self.package_dir, suffix) + + def test__packages_with_ns(self) -> None: + cases = [ + # Namespace package with py.typed + ("ns_pkg_typed", self.path("ns_pkg_typed")), + ("ns_pkg_typed.a", self.path("ns_pkg_typed/a.py")), + ("ns_pkg_typed.b", self.path("ns_pkg_typed/b")), + ("ns_pkg_typed.b.c", self.path("ns_pkg_typed/b/c.py")), + ("ns_pkg_typed.a.a_var", ModuleNotFoundReason.NOT_FOUND), + + # Namespace package without py.typed + ("ns_pkg_untyped", ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS), + ("ns_pkg_untyped.a", ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS), + ("ns_pkg_untyped.b", ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS), + ("ns_pkg_untyped.b.c", ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS), + ("ns_pkg_untyped.a.a_var", ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS), + + # Regular package with py.typed + ("pkg_typed", self.path("pkg_typed/__init__.py")), + ("pkg_typed.a", self.path("pkg_typed/a.py")), + ("pkg_typed.b", self.path("pkg_typed/b/__init__.py")), + ("pkg_typed.b.c", self.path("pkg_typed/b/c.py")), + ("pkg_typed.a.a_var", ModuleNotFoundReason.NOT_FOUND), + + # Regular package without py.typed + ("pkg_untyped", ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS), + ("pkg_untyped.a", ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS), + ("pkg_untyped.b", ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS), + ("pkg_untyped.b.c", ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS), + ("pkg_untyped.a.a_var", ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS), + + # Top-level Python file in site-packages + ("standalone", ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS), + ("standalone.standalone_var", ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS), + + # Something that doesn't exist + ("does_not_exist", ModuleNotFoundReason.NOT_FOUND), + + # A regular package with an installed set of stubs + ("foo.bar", self.path("foo-stubs/bar.pyi")), + + # A regular, non-site-packages module + ("a", os.path.join(data_path, "pkg1/a.py")), + ] + for module, expected in cases: + template = "Find(" + module + ") got {}; expected {}" + + actual = self.fmc_ns.find_module(module) + assert_equal(actual, expected, template) + + def test__packages_without_ns(self) -> None: + cases = [ + # Namespace package with py.typed + ("ns_pkg_typed", ModuleNotFoundReason.NOT_FOUND), + ("ns_pkg_typed.a", ModuleNotFoundReason.NOT_FOUND), + ("ns_pkg_typed.b", ModuleNotFoundReason.NOT_FOUND), + ("ns_pkg_typed.b.c", ModuleNotFoundReason.NOT_FOUND), + ("ns_pkg_typed.a.a_var", ModuleNotFoundReason.NOT_FOUND), + + # Namespace package without py.typed + ("ns_pkg_untyped", ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS), + ("ns_pkg_untyped.a", ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS), + ("ns_pkg_untyped.b", ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS), + ("ns_pkg_untyped.b.c", ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS), + ("ns_pkg_untyped.a.a_var", ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS), + + # Regular package with py.typed + ("pkg_typed", self.path("pkg_typed/__init__.py")), + ("pkg_typed.a", self.path("pkg_typed/a.py")), + ("pkg_typed.b", self.path("pkg_typed/b/__init__.py")), + ("pkg_typed.b.c", self.path("pkg_typed/b/c.py")), + ("pkg_typed.a.a_var", ModuleNotFoundReason.NOT_FOUND), + + # Regular package without py.typed + ("pkg_untyped", ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS), + ("pkg_untyped.a", ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS), + ("pkg_untyped.b", ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS), + ("pkg_untyped.b.c", ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS), + ("pkg_untyped.a.a_var", ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS), + + # Top-level Python file in site-packages + ("standalone", ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS), + ("standalone.standalone_var", ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS), + + # Something that doesn't exist + ("does_not_exist", ModuleNotFoundReason.NOT_FOUND), + + # A regular package with an installed set of stubs + ("foo.bar", self.path("foo-stubs/bar.pyi")), + + # A regular, non-site-packages module + ("a", os.path.join(data_path, "pkg1/a.py")), + ] + for module, expected in cases: + template = "Find(" + module + ") got {}; expected {}" + + actual = self.fmc_nons.find_module(module) + assert_equal(actual, expected, template) diff --git a/test-data/packages/modulefinder-site-packages/foo-stubs/__init__.pyi b/test-data/packages/modulefinder-site-packages/foo-stubs/__init__.pyi new file mode 100644 index 0000000000000..e69de29bb2d1d diff --git a/test-data/packages/modulefinder-site-packages/foo-stubs/bar.pyi b/test-data/packages/modulefinder-site-packages/foo-stubs/bar.pyi new file mode 100644 index 0000000000000..bf896e8cdfa37 --- /dev/null +++ b/test-data/packages/modulefinder-site-packages/foo-stubs/bar.pyi @@ -0,0 +1 @@ +bar_var: str \ No newline at end of file diff --git a/test-data/packages/modulefinder-site-packages/foo/__init__.py b/test-data/packages/modulefinder-site-packages/foo/__init__.py new file mode 100644 index 0000000000000..e69de29bb2d1d diff --git a/test-data/packages/modulefinder-site-packages/foo/bar.py b/test-data/packages/modulefinder-site-packages/foo/bar.py new file mode 100644 index 0000000000000..a1c3b50eeeab8 --- /dev/null +++ b/test-data/packages/modulefinder-site-packages/foo/bar.py @@ -0,0 +1 @@ +bar_var = "bar" \ No newline at end of file diff --git a/test-data/packages/modulefinder-site-packages/ns_pkg_typed/a.py b/test-data/packages/modulefinder-site-packages/ns_pkg_typed/a.py new file mode 100644 index 0000000000000..9d71311c4d826 --- /dev/null +++ b/test-data/packages/modulefinder-site-packages/ns_pkg_typed/a.py @@ -0,0 +1 @@ +a_var = "a" \ No newline at end of file diff --git a/test-data/packages/modulefinder-site-packages/ns_pkg_typed/b/c.py b/test-data/packages/modulefinder-site-packages/ns_pkg_typed/b/c.py new file mode 100644 index 0000000000000..003a29a2ef670 --- /dev/null +++ b/test-data/packages/modulefinder-site-packages/ns_pkg_typed/b/c.py @@ -0,0 +1 @@ +c_var = "c" \ No newline at end of file diff --git a/test-data/packages/modulefinder-site-packages/ns_pkg_typed/py.typed b/test-data/packages/modulefinder-site-packages/ns_pkg_typed/py.typed new file mode 100644 index 0000000000000..e69de29bb2d1d diff --git a/test-data/packages/modulefinder-site-packages/ns_pkg_untyped/a.py b/test-data/packages/modulefinder-site-packages/ns_pkg_untyped/a.py new file mode 100644 index 0000000000000..9d71311c4d826 --- /dev/null +++ b/test-data/packages/modulefinder-site-packages/ns_pkg_untyped/a.py @@ -0,0 +1 @@ +a_var = "a" \ No newline at end of file diff --git a/test-data/packages/modulefinder-site-packages/ns_pkg_untyped/b/c.py b/test-data/packages/modulefinder-site-packages/ns_pkg_untyped/b/c.py new file mode 100644 index 0000000000000..003a29a2ef670 --- /dev/null +++ b/test-data/packages/modulefinder-site-packages/ns_pkg_untyped/b/c.py @@ -0,0 +1 @@ +c_var = "c" \ No newline at end of file diff --git a/test-data/packages/modulefinder-site-packages/pkg_typed/__init__.py b/test-data/packages/modulefinder-site-packages/pkg_typed/__init__.py new file mode 100644 index 0000000000000..88ed99fb525e5 --- /dev/null +++ b/test-data/packages/modulefinder-site-packages/pkg_typed/__init__.py @@ -0,0 +1 @@ +pkg_typed_var = "pkg_typed" \ No newline at end of file diff --git a/test-data/packages/modulefinder-site-packages/pkg_typed/a.py b/test-data/packages/modulefinder-site-packages/pkg_typed/a.py new file mode 100644 index 0000000000000..9d71311c4d826 --- /dev/null +++ b/test-data/packages/modulefinder-site-packages/pkg_typed/a.py @@ -0,0 +1 @@ +a_var = "a" \ No newline at end of file diff --git a/test-data/packages/modulefinder-site-packages/pkg_typed/b/__init__.py b/test-data/packages/modulefinder-site-packages/pkg_typed/b/__init__.py new file mode 100644 index 0000000000000..de0052886c57b --- /dev/null +++ b/test-data/packages/modulefinder-site-packages/pkg_typed/b/__init__.py @@ -0,0 +1 @@ +b_var = "b" \ No newline at end of file diff --git a/test-data/packages/modulefinder-site-packages/pkg_typed/b/c.py b/test-data/packages/modulefinder-site-packages/pkg_typed/b/c.py new file mode 100644 index 0000000000000..003a29a2ef670 --- /dev/null +++ b/test-data/packages/modulefinder-site-packages/pkg_typed/b/c.py @@ -0,0 +1 @@ +c_var = "c" \ No newline at end of file diff --git a/test-data/packages/modulefinder-site-packages/pkg_typed/py.typed b/test-data/packages/modulefinder-site-packages/pkg_typed/py.typed new file mode 100644 index 0000000000000..e69de29bb2d1d diff --git a/test-data/packages/modulefinder-site-packages/pkg_untyped/__init__.py b/test-data/packages/modulefinder-site-packages/pkg_untyped/__init__.py new file mode 100644 index 0000000000000..c7ff39c111791 --- /dev/null +++ b/test-data/packages/modulefinder-site-packages/pkg_untyped/__init__.py @@ -0,0 +1 @@ +pkg_untyped_var = "pkg_untyped" \ No newline at end of file diff --git a/test-data/packages/modulefinder-site-packages/pkg_untyped/a.py b/test-data/packages/modulefinder-site-packages/pkg_untyped/a.py new file mode 100644 index 0000000000000..9d71311c4d826 --- /dev/null +++ b/test-data/packages/modulefinder-site-packages/pkg_untyped/a.py @@ -0,0 +1 @@ +a_var = "a" \ No newline at end of file diff --git a/test-data/packages/modulefinder-site-packages/pkg_untyped/b/__init__.py b/test-data/packages/modulefinder-site-packages/pkg_untyped/b/__init__.py new file mode 100644 index 0000000000000..de0052886c57b --- /dev/null +++ b/test-data/packages/modulefinder-site-packages/pkg_untyped/b/__init__.py @@ -0,0 +1 @@ +b_var = "b" \ No newline at end of file diff --git a/test-data/packages/modulefinder-site-packages/pkg_untyped/b/c.py b/test-data/packages/modulefinder-site-packages/pkg_untyped/b/c.py new file mode 100644 index 0000000000000..003a29a2ef670 --- /dev/null +++ b/test-data/packages/modulefinder-site-packages/pkg_untyped/b/c.py @@ -0,0 +1 @@ +c_var = "c" \ No newline at end of file diff --git a/test-data/packages/modulefinder-site-packages/standalone.py b/test-data/packages/modulefinder-site-packages/standalone.py new file mode 100644 index 0000000000000..35b38168f25eb --- /dev/null +++ b/test-data/packages/modulefinder-site-packages/standalone.py @@ -0,0 +1 @@ +standalone_var = "standalone" \ No newline at end of file diff --git a/test-data/unit/check-errorcodes.test b/test-data/unit/check-errorcodes.test index 89e7770045519..6eb0751c8df80 100644 --- a/test-data/unit/check-errorcodes.test +++ b/test-data/unit/check-errorcodes.test @@ -493,10 +493,9 @@ if int() is str(): # E: Non-overlapping identity check (left operand type: "int [builtins fixtures/primitives.pyi] [case testErrorCodeMissingModule] -from defusedxml import xyz # E: No library stub file for module 'defusedxml' [import] \ - # N: (Stub files are from https://github.com/python/typeshed) -from nonexistent import foobar # E: Cannot find implementation or library stub for module named 'nonexistent' [import] \ - # N: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports +from defusedxml import xyz # E: Cannot find implementation or library stub for module named 'defusedxml' [import] \ + # N: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports +from nonexistent import foobar # E: Cannot find implementation or library stub for module named 'nonexistent' [import] import nonexistent2 # E: Cannot find implementation or library stub for module named 'nonexistent2' [import] from nonexistent3 import * # E: Cannot find implementation or library stub for module named 'nonexistent3' [import] from pkg import bad # E: Module 'pkg' has no attribute 'bad' [attr-defined] diff --git a/test-data/unit/semanal-errors.test b/test-data/unit/semanal-errors.test index 042b39658df0d..b7cb7fd7706a6 100644 --- a/test-data/unit/semanal-errors.test +++ b/test-data/unit/semanal-errors.test @@ -1283,12 +1283,6 @@ x = 1 y = 1 [out] -[case testMissingStubForThirdPartyModule] -import __dummy_third_party1 -[out] -main:1: error: No library stub file for module '__dummy_third_party1' -main:1: note: (Stub files are from https://github.com/python/typeshed) - [case testMissingStubForStdLibModule] import __dummy_stdlib1 [out]