Release/20.2.1

.github/ISSUE_TEMPLATE/bug-report.md

@@ -4,7 +4,7 @@ about: Create a report to help us improve
 ---
 
 <!--
-If you're reporting an issue for `--unstable-feature=resolver`, use the "Dependency resolver failures / errors" template instead.
+If you're reporting an issue for `--use-feature=2020-resolver`, use the "Dependency resolver failures / errors" template instead.
 -->
 
 **Environment**

NEWS.rst

@@ -9,6 +9,34 @@
 
 .. towncrier release notes start
 
+20.2.1 (2020-08-04)
+===================
+
+Features
+--------
+
+- Ignore require-virtualenv in ``pip list`` (`#8603 <https://github.com/pypa/pip/issues/8603>`_)
+
+Bug Fixes
+---------
+
+- Correctly find already-installed distributions with dot (``.``) in the name
+  and uninstall them when needed. (`#8645 <https://github.com/pypa/pip/issues/8645>`_)
+- Trace a better error message on installation failure due to invalid ``.data``
+  files in wheels. (`#8654 <https://github.com/pypa/pip/issues/8654>`_)
+- Fix SVN version detection for alternative SVN distributions. (`#8665 <https://github.com/pypa/pip/issues/8665>`_)
+- New resolver: Correctly include the base package when specified with extras
+  in ``--no-deps`` mode. (`#8677 <https://github.com/pypa/pip/issues/8677>`_)
+- Use UTF-8 to handle ZIP archive entries on Python 2 according to PEP 427, so
+  non-ASCII paths can be resolved as expected. (`#8684 <https://github.com/pypa/pip/issues/8684>`_)
+
+Improved Documentation
+----------------------
+
+- Add details on old resolver deprecation and removal to migration documentation. (`#8371 <https://github.com/pypa/pip/issues/8371>`_)
+- Fix feature flag name in docs. (`#8660 <https://github.com/pypa/pip/issues/8660>`_)
+
+
 20.2 (2020-07-29)
 =================
 
@@ -43,6 +71,7 @@ Features
   break. More details about how to test and migrate, and how to report
   issues, at :ref:`Resolver changes 2020` . Maintainers are preparing to
   release pip 20.3, with the new resolver on by default, in October. (`#6536 <https://github.com/pypa/pip/issues/6536>`_)
+- Introduce a new ResolutionImpossible error, raised when pip encounters un-satisfiable dependency conflicts (`#8546 <https://github.com/pypa/pip/issues/8546>`_, `#8377 <https://github.com/pypa/pip/issues/8377>`_)
 - Add a subcommand ``debug`` to ``pip config`` to list available configuration sources and the key-value pairs defined in them. (`#6741 <https://github.com/pypa/pip/issues/6741>`_)
 - Warn if index pages have unexpected content-type (`#6754 <https://github.com/pypa/pip/issues/6754>`_)
 - Allow specifying ``--prefer-binary`` option in a requirements file (`#7693 <https://github.com/pypa/pip/issues/7693>`_)
@@ -92,7 +121,7 @@ Improved Documentation
 - Fix pip config docstring so that the subcommands render correctly in the docs (`#8072 <https://github.com/pypa/pip/issues/8072>`_)
 - replace links to the old pypa-dev mailing list with https://mail.python.org/mailman3/lists/distutils-sig.python.org/ (`#8353 <https://github.com/pypa/pip/issues/8353>`_)
 - Fix example for defining multiple values for options which support them (`#8373 <https://github.com/pypa/pip/issues/8373>`_)
-- Add documentation that helps the user fix dependency conflicts (`#8459 <https://github.com/pypa/pip/issues/8459>`_)
+- Add documentation for the ResolutionImpossible error that helps the user fix dependency conflicts (`#8459 <https://github.com/pypa/pip/issues/8459>`_)
 - Add feature flags to docs (`#8512 <https://github.com/pypa/pip/issues/8512>`_)
 - Document how to install package extras from git branch and source distributions. (`#8576 <https://github.com/pypa/pip/issues/8576>`_)
 

docs/html/development/release-process.rst

@@ -2,6 +2,7 @@
 Release process
 ===============
 
+.. _`Release Cadence`:
 
 Release Cadence
 ===============
@@ -72,6 +73,8 @@ only bugs will be considered, and merged (subject to normal review processes).
 Note that there may be delays due to the lack of developer resources for
 reviewing such pull requests.
 
+.. _`Feature Flags`:
+
 Feature Flags
 =============
 

docs/html/user_guide.rst

@@ -783,7 +783,7 @@ specified packages due to conflicting dependencies (a
 ``ResolutionImpossible`` error).
 
 This documentation is specific to the new resolver, which you can use
-with the flag ``--unstable-feature=resolver``.
+with the flag ``--use-feature=2020-resolver``.
 
 Understanding your error message
 --------------------------------
@@ -945,7 +945,7 @@ Unfortunately, **the pip team cannot provide support for individual
 dependency conflict errors**. Please *only* open a ticket on the `pip
 issue tracker`_ if you believe that your problem has exposed a bug in pip.
 
-.. _dependency hell: https://en.wikipedia.org/wiki/Dependency_hell>
+.. _dependency hell: https://en.wikipedia.org/wiki/Dependency_hell
 .. _Awesome Python: https://python.libhunt.com/
 .. _Python user Discourse: https://discuss.python.org/c/users/7
 .. _Python user forums: https://www.python.org/community/forums/
@@ -1029,13 +1029,13 @@ Changes to the pip dependency resolver in 20.2 (2020)
 =====================================================
 
 pip 20.1 included an alpha version of the new resolver (hidden behind
-an optional ``--unstable-feature=resolver`` flag). pip 20.2 includes a
-robust beta of the new resolver (hidden behind an optional
-``--use-feature=2020-resolver`` flag) that we encourage you to
-test. We will continue to improve the pip dependency resolver in
-response to testers' feedback. Please give us feedback through the
-`resolver testing survey`_. This will help us prepare to release pip
-20.3, with the new resolver on by default, in October.
+an optional ``--unstable-feature=resolver`` flag). pip 20.2 removes
+that flag, and includes a robust beta of the new resolver (hidden
+behind an optional ``--use-feature=2020-resolver`` flag) that we
+encourage you to test. We will continue to improve the pip dependency
+resolver in response to testers' feedback. Please give us feedback
+through the `resolver testing survey`_. This will help us prepare to
+release pip 20.3, with the new resolver on by default, in October.
 
 Watch out for
 -------------
@@ -1223,22 +1223,42 @@ Specific things we'd love to get feedback on:
 
 Please let us know through the `resolver testing survey`_.
 
+Deprecation timeline
+--------------------
+
+We plan for the resolver changeover to proceed as follows, using
+:ref:`Feature Flags` and following our :ref:`Release Cadence`:
+
+*    pip 20.2: a beta of the new resolver is available, opt-in, using
+     the flag ``--use-feature=2020-resolver``. pip defaults to
+     legacy behavior.
+
+*    pip 20.3: pip defaults to the new resolver, but a user can opt-out
+     and choose the old resolver behavior, using the flag
+     ``--use-deprecated=legacy-resolver``.
+
+*    pip 21.0: pip uses new resolver, and the old resolver is no longer
+     available.
+
+Since this work will not change user-visible behavior described in the
+pip documentation, this change is not covered by the :ref:`Deprecation
+Policy`.
+
 Context and followup
 --------------------
 
 As discussed in `our announcement on the PSF blog`_, the pip team are
 in the process of developing a new "dependency resolver" (the part of
-pip that works out what to install based on your requirements). Since
-this work will not change user-visible behavior described in the pip
-documentation, this change is not covered by the :ref:`Deprecation
-Policy`.
+pip that works out what to install based on your requirements).
 
 We're tracking our rollout in :issue:`6536` and you can watch for
-announcements on the `low-traffic packaging announcements list`_.
+announcements on the `low-traffic packaging announcements list`_ and
+`the official Python blog`_.
 
 .. _freeze: https://pip.pypa.io/en/latest/reference/pip_freeze/
 .. _resolver testing survey: https://tools.simplysecure.org/survey/index.php?r=survey/index&sid=989272&lang=en
 .. _our announcement on the PSF blog: http://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html
 .. _tensorflow: https://pypi.org/project/tensorflow/
 .. _low-traffic packaging announcements list: https://mail.python.org/mailman3/lists/pypi-announce.python.org/
 .. _our survey on upgrades that create conflicts: https://docs.google.com/forms/d/e/1FAIpQLSeBkbhuIlSofXqCyhi3kGkLmtrpPOEBwr6iJA6SzHdxWKfqdA/viewform
+.. _the official Python blog: https://blog.python.org/

src/pip/__init__.py

@@ -4,7 +4,7 @@
     from typing import List, Optional
 
 
-__version__ = "20.2"
+__version__ = "20.3.dev0"
 
 
 def main(args=None):

src/pip/_internal/commands/list.py

@@ -39,6 +39,7 @@ class ListCommand(IndexGroupCommand):
     Packages are listed in a case-insensitive sorted order.
     """
 
+    ignore_require_venv = True
     usage = """
       %prog [options]"""
 

src/pip/_internal/network/lazy_wheel.py

@@ -109,8 +109,10 @@ def read(self, size=-1):
         all bytes until EOF are returned.  Fewer than
         size bytes may be returned if EOF is reached.
         """
+        download_size = max(size, self._chunk_size)
         start, length = self.tell(), self._length
-        stop = start + size if 0 <= size <= length-start else length
+        stop = length if size < 0 else min(start+download_size, length)
+        start = max(0, stop-download_size)
         self._download(start, stop-1)
         return self._file.read(size)
 

src/pip/_internal/operations/install/wheel.py

@@ -78,6 +78,7 @@
         Union,
         cast,
     )
+    from zipfile import ZipInfo
 
     from pip._vendor.pkg_resources import Distribution
 
@@ -420,6 +421,15 @@ def __init__(self, src_record_path, dest_path, zip_file):
         self._zip_file = zip_file
         self.changed = False
 
+    def _getinfo(self):
+        # type: () -> ZipInfo
+        if not PY2:
+            return self._zip_file.getinfo(self.src_record_path)
+        # Python 2 does not expose a way to detect a ZIP's encoding, but the
+        # wheel specification (PEP 427) explicitly mandates that paths should
+        # use UTF-8, so we assume it is true.
+        return self._zip_file.getinfo(self.src_record_path.encode("utf-8"))
+
     def save(self):
         # type: () -> None
         # directory creation is lazy and after file filtering
@@ -439,11 +449,12 @@ def save(self):
         if os.path.exists(self.dest_path):
             os.unlink(self.dest_path)
 
-        with self._zip_file.open(self.src_record_path) as f:
+        zipinfo = self._getinfo()
+
+        with self._zip_file.open(zipinfo) as f:
             with open(self.dest_path, "wb") as dest:
                 shutil.copyfileobj(f, dest)
 
-        zipinfo = self._zip_file.getinfo(self.src_record_path)
         if zip_item_is_executable(zipinfo):
             set_extracted_file_to_default_mode_plus_executable(self.dest_path)
 
@@ -583,8 +594,28 @@ def data_scheme_file_maker(zip_file, scheme):
         def make_data_scheme_file(record_path):
             # type: (RecordPath) -> File
             normed_path = os.path.normpath(record_path)
-            _, scheme_key, dest_subpath = normed_path.split(os.path.sep, 2)
-            scheme_path = scheme_paths[scheme_key]
+            try:
+                _, scheme_key, dest_subpath = normed_path.split(os.path.sep, 2)
+            except ValueError:
+                message = (
+                    "Unexpected file in {}: {!r}. .data directory contents"
+                    " should be named like: '<scheme key>/<path>'."
+                ).format(wheel_path, record_path)
+                raise InstallationError(message)
+
+            try:
+                scheme_path = scheme_paths[scheme_key]
+            except KeyError:
+                valid_scheme_keys = ", ".join(sorted(scheme_paths))
+                message = (
+                    "Unknown scheme key used in {}: {} (for file {!r}). .data"
+                    " directory contents should be in subdirectories named"
+                    " with a valid scheme key ({})"
+                ).format(
+                    wheel_path, scheme_key, record_path, valid_scheme_keys
+                )
+                raise InstallationError(message)
+
             dest_path = os.path.join(scheme_path, dest_subpath)
             assert_no_path_traversal(scheme_path, dest_path)
             return ZipBackedFile(record_path, dest_path, zip_file)

src/pip/_internal/req/req_install.py

@@ -429,25 +429,13 @@ def check_if_exists(self, use_user_site):
         """
         if self.req is None:
             return
-        # get_distribution() will resolve the entire list of requirements
-        # anyway, and we've already determined that we need the requirement
-        # in question, so strip the marker so that we don't try to
-        # evaluate it.
-        no_marker = Requirement(str(self.req))
-        no_marker.marker = None
-
-        # pkg_resources uses the canonical name to look up packages, but
-        # the name passed passed to get_distribution is not canonicalized
-        # so we have to explicitly convert it to a canonical name
-        no_marker.name = canonicalize_name(no_marker.name)
-        try:
-            self.satisfied_by = pkg_resources.get_distribution(str(no_marker))
-        except pkg_resources.DistributionNotFound:
+        existing_dist = get_distribution(self.req.name)
+        if not existing_dist:
             return
-        except pkg_resources.VersionConflict:
-            existing_dist = get_distribution(
-                self.req.name
-            )
+
+        existing_version = existing_dist.parsed_version
+        if not self.req.specifier.contains(existing_version, prereleases=True):
+            self.satisfied_by = None
             if use_user_site:
                 if dist_in_usersite(existing_dist):
                     self.should_reinstall = True
@@ -461,11 +449,13 @@ def check_if_exists(self, use_user_site):
             else:
                 self.should_reinstall = True
         else:
-            if self.editable and self.satisfied_by:
+            if self.editable:
                 self.should_reinstall = True
                 # when installing editables, nothing pre-existing should ever
                 # satisfy
                 self.satisfied_by = None
+            else:
+                self.satisfied_by = existing_dist
 
     # Things valid for wheels
     @property

src/pip/_internal/resolution/resolvelib/base.py

@@ -69,8 +69,8 @@ def source_link(self):
         # type: () -> Optional[Link]
         raise NotImplementedError("Override in subclass")
 
-    def iter_dependencies(self):
-        # type: () -> Iterable[Optional[Requirement]]
+    def iter_dependencies(self, with_requires):
+        # type: (bool) -> Iterable[Optional[Requirement]]
         raise NotImplementedError("Override in subclass")
 
     def get_install_requirement(self):

src/pip/_internal/resolution/resolvelib/candidates.py

@@ -275,8 +275,10 @@ def _get_requires_python_specifier(self):
             return None
         return spec
 
-    def iter_dependencies(self):
-        # type: () -> Iterable[Optional[Requirement]]
+    def iter_dependencies(self, with_requires):
+        # type: (bool) -> Iterable[Optional[Requirement]]
+        if not with_requires:
+            return
         for r in self.dist.requires():
             yield self._factory.make_requirement_from_spec(str(r), self._ireq)
         python_dep = self._factory.make_requires_python_requirement(
@@ -420,8 +422,10 @@ def format_for_error(self):
         # type: () -> str
         return "{} {} (Installed)".format(self.name, self.version)
 
-    def iter_dependencies(self):
-        # type: () -> Iterable[Optional[Requirement]]
+    def iter_dependencies(self, with_requires):
+        # type: (bool) -> Iterable[Optional[Requirement]]
+        if not with_requires:
+            return
         for r in self.dist.requires():
             yield self._factory.make_requirement_from_spec(str(r), self._ireq)
 
@@ -519,10 +523,16 @@ def source_link(self):
         # type: () -> Optional[Link]
         return self.base.source_link
 
-    def iter_dependencies(self):
-        # type: () -> Iterable[Optional[Requirement]]
+    def iter_dependencies(self, with_requires):
+        # type: (bool) -> Iterable[Optional[Requirement]]
         factory = self.base._factory
 
+        # Add a dependency on the exact base
+        # (See note 2b in the class docstring)
+        yield factory.make_requirement_from_candidate(self.base)
+        if not with_requires:
+            return
+
         # The user may have specified extras that the candidate doesn't
         # support. We ignore any unsupported extras here.
         valid_extras = self.extras.intersection(self.base.dist.extras)
@@ -535,10 +545,6 @@ def iter_dependencies(self):
                 extra
             )
 
-        # Add a dependency on the exact base
-        # (See note 2b in the class docstring)
-        yield factory.make_requirement_from_candidate(self.base)
-
         for r in self.base.dist.requires(valid_extras):
             requirement = factory.make_requirement_from_spec(
                 str(r), self.base._ireq, valid_extras,
@@ -585,8 +591,8 @@ def format_for_error(self):
         # type: () -> str
         return "Python {}".format(self.version)
 
-    def iter_dependencies(self):
-        # type: () -> Iterable[Optional[Requirement]]
+    def iter_dependencies(self, with_requires):
+        # type: (bool) -> Iterable[Optional[Requirement]]
         return ()
 
     def get_install_requirement(self):

src/pip/_internal/resolution/resolvelib/factory.py

@@ -453,7 +453,7 @@ def describe_trigger(parent):
         logger.info(msg)
 
         return DistributionNotFound(
-            "ResolutionImpossible For help visit: "
-            "https://pip.pypa.io/en/stable/user_guide/"
+            "ResolutionImpossible: for help visit "
+            "https://pip.pypa.io/en/latest/user_guide/"
             "#fixing-conflicting-dependencies"
         )