pip3 broken?

[selfagency]

What did you expect to happen?

Python packages should have updated. Running pip3 list --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip3 install -U works without issue.

What actually happened?

―― 09:38:23 - pip3 ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
Requirement already satisfied: pip in ./Library/Python/3.9/lib/python/site-packages (22.2)
Collecting pip
  Using cached pip-22.2.1-py3-none-any.whl (2.0 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.2
    Uninstalling pip-22.2:
      Successfully uninstalled pip-22.2
  Rolling back uninstall of pip
  Moving to /Users/daniel/Library/Python/3.9/bin/
   from /Users/daniel/Library/Python/3.9/~in
  Moving to /Users/daniel/Library/Python/3.9/lib/python/site-packages/pip-22.2.dist-info/
   from /Users/daniel/Library/Python/3.9/lib/python/site-packages/~ip-22.2.dist-info
  Moving to /Users/daniel/Library/Python/3.9/lib/python/site-packages/pip/
   from /Users/daniel/Library/Python/3.9/lib/python/site-packages/~ip
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/daniel/Library/Python/3.9/lib/python/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "/Users/daniel/Library/Python/3.9/lib/python/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
    return func(self, options, args)
  File "/Users/daniel/Library/Python/3.9/lib/python/site-packages/pip/_internal/commands/install.py", line 461, in run
    installed = install_given_reqs(
  File "/Users/daniel/Library/Python/3.9/lib/python/site-packages/pip/_internal/req/__init__.py", line 73, in install_given_reqs
    requirement.install(
  File "/Users/daniel/Library/Python/3.9/lib/python/site-packages/pip/_internal/req/req_install.py", line 752, in install
    scheme = get_scheme(
  File "/Users/daniel/Library/Python/3.9/lib/python/site-packages/pip/_internal/locations/__init__.py", line 244, in get_scheme
    from . import _distutils
ImportError: cannot import name '_distutils' from 'pip._internal.locations' (/Users/daniel/Library/Python/3.9/lib/python/site-packages/pip/_internal/locations/__init__.py)

Retry? (y)es/(N)o/(s)hell/(q)uit%                                                                                                                                                                                                                                              

Additional Details

• Which operating system or Linux distribution are you using? Mac OS 12.5
• How did you install Topgrade? Homebrew
• Which version are you running? 9.0.1

This won't help because I ran it after I manually upgraded the packages:

09:40:59 AM ✦2 » topgrade --dry-run --only pip3

―― 09:41:18 - pip3 ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
Dry running: /opt/homebrew/bin/python3 -m pip install --upgrade --user pip

―― 09:41:18 - Summary ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
pip3: OK

Same here:

 2022-07-28T13:41:42.419Z DEBUG topgrade::config > Configuration at /Users/daniel/.config/topgrade.toml
 2022-07-28T13:41:42.419Z DEBUG topgrade::config > Loaded configuration: ConfigFile { pre_commands: Some({}), post_commands: None, commands: Some({}), git_repos: None, predefined_git_repos: None, disable: None, ignore_failures: None, remote_topgrades: None, remote_topgrade_path: None, ssh_arguments: None, git_arguments: None, tmux_arguments: None, set_title: None, display_time: None, assume_yes: None, yay_arguments: None, no_retry: None, run_in_tmux: None, cleanup: None, notify_each_step: None, accept_all_windows_updates: None, bashit_branch: None, only: None, composer: Some(Composer { self_update: None }), brew: Some(Brew { greedy_cask: None }), linux: Some(Linux { yay_arguments: None, arch_package_manager: None, show_arch_news: None, trizen_arguments: None, pikaur_arguments: None, dnf_arguments: None, apt_arguments: None, enable_tlmgr: None, redhat_distro_sync: None, rpm_ostree: None, emerge_sync_flags: None, emerge_update_flags: None }), git: Some(Git { max_concurrency: None, arguments: None, repos: None, pull_predefined: None }), windows: Some(Windows { accept_all_updates: None, self_rename: None, open_remotes_in_new_terminal: None, enable_winget: None }), npm: Some(NPM { use_sudo: None }), vim: None, firmware: Some(Firmware { upgrade: None }), vagrant: None, flatpak: Some(Flatpak { use_sudo: None }) }
 2022-07-28T13:41:42.419Z DEBUG topgrade         > Version: 9.0.1
 2022-07-28T13:41:42.419Z DEBUG topgrade         > OS: aarch64-apple-darwin
 2022-07-28T13:41:42.419Z DEBUG topgrade         > Args { inner: ["topgrade", "--only", "pip3", "-v"] }
 2022-07-28T13:41:42.419Z DEBUG topgrade         > Binary path: Ok("/opt/homebrew/bin/topgrade")
 2022-07-28T13:41:42.419Z DEBUG topgrade         > Self Update: false
 2022-07-28T13:41:42.419Z DEBUG topgrade::utils  > Detected "/opt/homebrew/bin/git" as "git"
 2022-07-28T13:41:42.420Z DEBUG topgrade::utils  > Cannot find "doas"
 2022-07-28T13:41:42.420Z DEBUG topgrade::utils  > Detected "/usr/bin/sudo" as "sudo"
 2022-07-28T13:41:42.420Z DEBUG topgrade::utils  > Detected "/usr/local/bin/pwsh" as "pwsh"
 2022-07-28T13:41:42.730Z TRACE topgrade::executor > Output of "/usr/local/bin/pwsh" "-NoProfile" "-Command" "Split-Path $profile": Output { status: ExitStatus(unix_wait_status(0)), stdout: "/Users/daniel/.config/powershell\n", stderr: "" }
 2022-07-28T13:41:42.730Z DEBUG topgrade::utils    > Path "/Users/daniel/.config/emacs" doesn't exist
 2022-07-28T13:41:42.730Z DEBUG topgrade::utils    > Path "/Users/daniel/.emacs.d" doesn't exist
 2022-07-28T13:41:42.730Z DEBUG topgrade::steps::git > /Users/daniel/.ideavimrc does not exists
 2022-07-28T13:41:42.730Z DEBUG topgrade::steps::git > /Users/daniel/.intellimacs does not exists
 2022-07-28T13:41:42.730Z DEBUG topgrade::steps::git > /Users/daniel/.config/.zshrc is a file. Checking /Users/daniel/.config
 2022-07-28T13:41:42.730Z DEBUG topgrade::steps::git > Checking if /Users/daniel/.config is a git repository
 2022-07-28T13:41:42.734Z TRACE topgrade::executor   > Output of "/opt/homebrew/bin/git" "rev-parse" "--show-toplevel": Output { status: ExitStatus(unix_wait_status(32768)), stdout: "", stderr: "fatal: not a git repository (or any of the parent directories): .git\n" }
 2022-07-28T13:41:42.734Z DEBUG topgrade::steps::git > Checking if /Users/daniel/.config/fish is a git repository
 2022-07-28T13:41:42.737Z TRACE topgrade::executor   > Output of "/opt/homebrew/bin/git" "rev-parse" "--show-toplevel": Output { status: ExitStatus(unix_wait_status(32768)), stdout: "", stderr: "fatal: not a git repository (or any of the parent directories): .git\n" }
 2022-07-28T13:41:42.737Z DEBUG topgrade::steps::git > /Users/daniel/Library/Application Support/openbox does not exists
 2022-07-28T13:41:42.737Z DEBUG topgrade::steps::git > /Users/daniel/Library/Application Support/bspwm does not exists
 2022-07-28T13:41:42.737Z DEBUG topgrade::steps::git > /Users/daniel/Library/Application Support/i3 does not exists
 2022-07-28T13:41:42.737Z DEBUG topgrade::steps::git > /Users/daniel/Library/Application Support/sway does not exists
 2022-07-28T13:41:42.737Z DEBUG topgrade::runner     > Step "pip3"
 2022-07-28T13:41:42.737Z DEBUG topgrade::utils      > Detected "/opt/homebrew/bin/python3" as "python3"
 2022-07-28T13:41:42.876Z TRACE topgrade::executor   > Output of "/opt/homebrew/bin/python3" "-m" "pip": Output { status: ExitStatus(unix_wait_status(0)), stdout: "\nUsage:   \n  /opt/homebrew/opt/[email protected]/bin/python3.9 -m pip <command> [options]\n\nCommands:\n  install                     Install packages.\n  download                    Download packages.\n  uninstall                   Uninstall packages.\n  freeze                      Output installed packages in requirements format.\n  inspect                     Inspect the python environment.\n  list                        List installed packages.\n  show                        Show information about installed packages.\n  check                       Verify installed packages have compatible dependencies.\n  config                      Manage local and global configuration.\n  search                      Search PyPI for packages.\n  cache                       Inspect and manage pip's wheel cache.\n  index                       Inspect information available from package indexes.\n  wheel                       Build wheels from your requirements.\n  hash                        Compute hashes of package archives.\n  completion                  A helper command used for command completion.\n  debug                       Show information useful for debugging.\n  help                        Show help for commands.\n\nGeneral Options:\n  -h, --help                  Show help.\n  --debug                     Let unhandled exceptions propagate outside the\n                              main subroutine, instead of logging them to\n                              stderr.\n  --isolated                  Run pip in an isolated mode, ignoring\n                              environment variables and user configuration.\n  --require-virtualenv        Allow pip to only run in a virtual environment;\n                              exit with an error otherwise.\n  -v, --verbose               Give more output. Option is additive, and can be\n                              used up to 3 times.\n  -V, --version               Show version and exit.\n  -q, --quiet                 Give less output. Option is additive, and can be\n                              used up to 3 times (corresponding to WARNING,\n                              ERROR, and CRITICAL logging levels).\n  --log <path>                Path to a verbose appending log.\n  --no-input                  Disable prompting for input.\n  --proxy <proxy>             Specify a proxy in the form\n                              scheme://[user:passwd@]proxy.server:port.\n  --retries <retries>         Maximum number of retries each connection should\n                              attempt (default 5 times).\n  --timeout <sec>             Set the socket timeout (default 15 seconds).\n  --exists-action <action>    Default action when a path already exists:\n                              (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort.\n  --trusted-host <hostname>   Mark this host or host:port pair as trusted,\n                              even though it does not have valid or any HTTPS.\n  --cert <path>               Path to PEM-encoded CA certificate bundle. If\n                              provided, overrides the default. See 'SSL\n                              Certificate Verification' in pip documentation\n                              for more information.\n  --client-cert <path>        Path to SSL client certificate, a single file\n                              containing the private key and the certificate\n                              in PEM format.\n  --cache-dir <dir>           Store the cache data in <dir>.\n  --no-cache-dir              Disable the cache.\n  --disable-pip-version-check\n                              Don't periodically check PyPI to determine\n                              whether a new version of pip is available for\n                              download. Implied with --no-index.\n  --no-color                  Suppress colored output.\n  --no-python-version-warning\n                              Silence deprecation warnings for upcoming\n                              unsupported Pythons.\n  --use-feature <feature>     Enable new functionality, that may be backward\n                              incompatible.\n  --use-deprecated <feature>  Enable deprecated functionality, that will be\n                              removed in the future.\n", stderr: "" }

―― 09:41:42 - pip3 ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
 2022-07-28T13:41:42.876Z DEBUG topgrade::executor   > Running "/opt/homebrew/bin/python3" "-m" "pip" "install" "--upgrade" "--user" "pip"
Requirement already satisfied: pip in /opt/homebrew/lib/python3.9/site-packages (22.2.1)

―― 09:41:43 - Summary ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
pip3: OK
 2022-07-28T13:41:43.361Z DEBUG topgrade::terminal   > Desktop notification: Topgrade finished successfully

[MCOfficer]

Looks like a pip issue: pypa/pip#11319

[tos-kamiya]

I experienced the same error.

I understand that it is a pip error, but if topgrade would update pip first (python3 -m pip install --upgrade pip), it would prevent the users from experiencing this error in the future.

Requirement already satisfied: pip in ./.local/lib/python3.8/site-packages (22.2)
Collecting pip
  Using cached pip-22.2.1-py3-none-any.whl (2.0 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.2
    Uninstalling pip-22.2:
      Successfully uninstalled pip-22.2
  Rolling back uninstall of pip
  Moving to /home/toshihiro/.local/bin/pip
   from /tmp/pip-uninstall-37ltymkz/pip
  Moving to /home/toshihiro/.local/bin/pip3
   from /tmp/pip-uninstall-37ltymkz/pip3
  Moving to /home/toshihiro/.local/bin/pip3.8
   from /tmp/pip-uninstall-37ltymkz/pip3.8
  Moving to /home/toshihiro/.local/lib/python3.8/site-packages/pip-22.2.dist-info/
   from /home/toshihiro/.local/lib/python3.8/site-packages/~ip-22.2.dist-info
  Moving to /home/toshihiro/.local/lib/python3.8/site-packages/pip/
   from /home/toshihiro/.local/lib/python3.8/site-packages/~ip
ERROR: Exception:
Traceback (most recent call last):
  File "/home/toshihiro/.local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "/home/toshihiro/.local/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
    return func(self, options, args)
  File "/home/toshihiro/.local/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 461, in run
    installed = install_given_reqs(
  File "/home/toshihiro/.local/lib/python3.8/site-packages/pip/_internal/req/__init__.py", line 73, in install_given_reqs
    requirement.install(
  File "/home/toshihiro/.local/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 752, in install
    scheme = get_scheme(
  File "/home/toshihiro/.local/lib/python3.8/site-packages/pip/_internal/locations/__init__.py", line 244, in get_scheme
    from . import _distutils
ImportError: cannot import name '_distutils' from 'pip._internal.locations' (/home/toshihiro/.local/lib/python3.8/site-packages/pip/_internal/locations/__init__.py)

[MCOfficer]

Are you sure? By my understanding, upgrading pip first would just make it fail sooner. I don't think it's within topgrade's scope to account for unexpected behavior of tools like pip.

[tos-kamiya]

Since topgrade is doing carefully rustup before the cargo update, it would be nice to have the same behavior for pip, by updating the pip itself before the pip update. However, I also understand the policy of not addressing it because it is a pip-specific error, as you mentioned.

[MCOfficer]

By my understanding, rustup runs so any following cargo builds don't fail due to an outdated toolchain. If the same can be said about pip (i wouldn't know), that's a valid argument. But this issue is more akin to rustup update failing with a segfault - at which point we run it is not really relevant, it's an upstream issue either way.

[tos-kamiya]

In my experience, there have been several times when the latest version of some library failed to install because the pip was out of date. So, in my mind, if pip fails, the right thing to do is to upgrade pip itself.

However, it seems that the upgrade of pip itself can cause problems in a Windows environment, although I don't use Windows very often.
pypa/pip#9395

So, I reconsider that it may be a hasty decision to include the upgrade function of pip itself in topgrade as a default behavior.

[joschi]

Until this has been fixed in pip 22.2.2, you can update pip manually using this command (from pypa/pip#11319 (comment)).

python -m pip install --user --upgrade https://github.com/pypa/pip/archive/refs/tags/22.2.1.zip
# python3 -m pip install --user --upgrade https://github.com/pypa/pip/archive/refs/tags/22.2.1.zip