pip v10 breaks Debian/Ubuntu pip3 command

[iamarcel]

Maintainer note: Anyone that still gets this issue please see #5599.

---

• Pip version: 10.0.0
• Python version: 3.5.2
• Operating system: Ubuntu 16.04 (EDIT: tested on debian:9.4 too, same thing happens)

Description:

When upgrading pip (to v10) on at least Ubuntu 16.04, the pip3 command stops working ("cannot import main", see below). This is on a fresh install.

What I've run:

(Note that I've stripped out all the apt output etc., since I think it's not needed here. Let me know if you still want it!)

me@host$ sudo docker run -it ubuntu:xenial

root@container# apt update && apt install python3-pip

root@container# pip3 --version
pip 8.1.1 from /usr/lib/python3/dist-packages (python 3.5)

root@container# pip3 install --upgrade pip
Collecting pip
  Downloading pip-10.0.0-py2.py3-none-any.whl (1.3MB)
    100% |################################| 1.3MB 1.4MB/s 
Installing collected packages: pip
  Found existing installation: pip 8.1.1
    Not uninstalling pip at /usr/lib/python3/dist-packages, outside environment /usr
Successfully installed pip-10.0.0

root@container# pip --version
pip 10.0.0 from /usr/local/lib/python3.5/dist-packages/pip (python 3.5)

root@container# pip3 --version
Traceback (most recent call last):
  File "/usr/bin/pip3", line 9, in <module>
    from pip import main
ImportError: cannot import name 'main'

root@container# cat /usr/bin/pip3
#!/usr/bin/python3
# GENERATED BY DEBIAN

import sys

# Run the main entry point, similarly to how setuptools does it, but because
# we didn't install the actual entry point from setup.py, don't use the
# pkg_resources API.
from pip import main
if __name__ == '__main__':
    sys.exit(main())

Not sure if this is something that should be fixed on the pip side or on the Debian side.

[RonnyPfannschmidt]

thats a debian issue

[RonnyPfannschmidt]

on extra note - replacing the system pip using pip is always an act of system-vandalism where the one inflicting it is responsible for the fallout

[pfmoore]

I'd suggest you should wait for a proper apt packaged copy of pip 10 from Debian - as @RonnyPfannschmidt says, you shouldn't be using pip to upgrade your system packages...

Looks like the Debian pip3 script uses pip's internals, so getting a pip10 compatible fix is definitely up to them (and I'd fully expect them to wait on releasing their pip10 package until they have that sorted).

[fake-name]

on extra note - replacing the system pip using pip is always an act of system-vandalism where the one inflicting it is responsible for the fallout

Convince the debian/ubuntu people to not vendor and then let rot half their packages, and that'd be a valid argument.

[pradyunsg]

You can use virtualenv or venv to isolate yourself from the system pip installation.

You should not be modifying the package-manager managed files (here the system pip installation) -- I think they don't expect users to modify things -- it is likely not supported by Debian. It's bound to cause issues like this.

[slabua]

Same issue on Fedora, too.

[di]

Perhaps there should be a "beta" channel, or some similar mechanism for people to do more testing prior to release, rather then just dumping a broken version on pypi and causing everyone's builds to explode.

@fake-name There were two pre-releases made:

• 10.0.0b1 was released on Mar 31st
• 10.0.0b2 was released on April 2nd

[RonnyPfannschmidt]

@fake-name additionally the general suggestion for usage on all distros is - use a virtualenv, don't vandalize the system, and that works - people just font follow it and then wonder when stuff breaks and blame pip

there has been plenty of manual and automated testing with virtualenvs which works

also in a virtualenv at least there should be no debian made pip3 command - so what the heck are you talking about wrt this breaking in virtualenvs - please provide enough data to actually verify instead of whining about breakages without providing the data needed to verify them

pip is volunteer driven, not a company with dozens of employees

[fake-name]

Deleted. was confusing this with #5220

I'm a derp.

[iamarcel]

Thanks, hadn't realized that replacing the system pip was a bad idea but it makes sense. However, it would be good UX if pip would not nag about upgrading in that case. Would that be possible? I reckon a lot of people (including me) would just do whatever "the thing" asks them to do.

[RonnyPfannschmidt]

@fake-name thanks for following up - its surprisingly common to mismatch the detail issue when a major release has multi-faceted impact and part of it tries to ruin you day

[pfmoore]

it would be good UX if pip would not nag about upgrading in that case

Distribution vendors could certainly patch pip to remove that warning (or better, replace it with a similar check against the system packages) along with the other patches they make. I'm not sure how base pip could detect that it's running from a system packaged installation without co-operation from the distribution, but if there's a way to do so that's something we could consider (but be aware that in my experience we get as much negative feedback from getting such heuristics wrong as we do from not including heuristics at all...)

[gsemet]

Always prefer "python3 -m pip" over pip3" or even better "/usr/bin/env python3 -m pip" it is safer and allow to avoid this issue with pip10

[standag]

We solved this issue by clear hash in bash:

$ hash -d pip

Or in dash (sh):

$ hash -r pip

[qacollective]

Also hit this issue while building docker images.

@RonnyPfannschmidt says "replacing the system pip using pip is always an act of system-vandalism where the one inflicting it is responsible for the fallout", which has 6 thumbs up. I find this to be an especially obtuse comment given that I instructed to do so by pip itself:

You are using pip version 8.1.1, however version 10.0.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

If there is some validity to that comment, then the creators of pip should remove this message and I would encourage @RonnyPfannschmidt to raise an issue to that effect and make his case.

[fake-name]

@qacollective - I think the argument here is that the distro's have taken pip, modified it, and re-packaged it into their repositories. As such, it's not really Pypi's fault the message is still there.

Most of this is because a bunch of the distros try really really hard to repackage everything into their own package repositories. Mostly, the things then get left to rot.

Personally, at least for python on ubuntu, I wish they'd quit it. The version of basically every python package in apt ranges from really, really old to fossilized. Apt is basically useless for python, IMHO.

---

FWIW, I tend to find the best option is to never install the distro pip in the first place, but instead install it manually via get-pip.py. That way, you don't have issues with the platform package manager knowing about only some of the python packages.

[gsemet]

Always use --user to avoid killing your system

/usr/bin/env python3 -m pip intall --user --upgrade pip

Should handle most of buggy case and result in the right version of pip being installed in ˜/.local/bin.

[gabormay]

I can confirm @standag's solution is working.

A bit of background: After the upgrade (pip install -U pip) on a vanilla ubuntu 16.04 (AWS AMI) one gets to the following situation:
$PATH=..:/usr/local/bin:...:/usr/bin:...
/usr/bin/pip is still the old/'oem' version (broken)
/usr/local/bin/pip is the new v10 script (works fine when invoked directly)

Even though the proper pip version preceeds the broken one in the PATH, bash still remembers the old one, so when you invoke it just as 'pip', you'll get the old, broken one running. hash-d pip or hash -r solves the issue.

[pradyunsg]

First, some notes about what happened here on Debian/Ubuntu (and likely a few more Linux Distros):

• pip does not support the use of it's internals by importing it. More on that in the documentation here.
• Debian (hence Ubuntu) does not support modifying their package manager managed files using something, that well, isn't their package manager.

This issue is caused by, well, both being violated in a way.

• Debian uses the pip's internal methods (which no longer work due to a reorganization of pip's internals). Debian's assuming here that the pip version in their repositories is the one that would be installed.
• Running pip install --upgrade pip, as root, without any other parameters modifies files that are supposed to managed by apt, which breaks the script by Debian.

Some general tips on Linux:

• It's a good habit to use --user whenever outside a venv.

  pip install --upgrade --user pip

• Never run pip with sudo unless you know what you're doing.

---

What's the workaround?

@standag's solution is useful when this is caused by bash's caching of executables.

hash -r pip # or hash -d pip

If you've modified your OS package manager's installation of pip (eg by using sudo pip) and python -m pip is still working, one workaround is to uninstall the pip installed version and reinstall the package manager installed version.

python -m pip uninstall pip  # this might need sudo
sudo apt install --reinstall python-pip

If you're not on Debian/Ubuntu and pip broke for you, try running:

python -m pip install --force-reinstall pip

If the above this doesn't resolve your problems, please file a new issue.

---

[edited by @pradyunsg: make it more relevant for linking everyone with similar issues to this comment; update the suggestion to include uninstalling/reinstalling workaround]

[ThinkDigitalSoftware]

What about solving it outside docker? It's broken in my regular system and the hash command didn't recognize pip.
thinkdigital@thinkdigital-HP-Spectre-x360-Convertible:~$ hash -d pip bash: hash: pip: not found

[ThinkDigitalSoftware]

I found pip3, version 9.0.1 installed in a virtualenv from a project and copied it to my /usr/bin and it works again. Here are the contents of the pip3 executable for those who would like to fix it themselves as well.

# -*- coding: utf-8 -*-
import re
import sys

from pip import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

I'm sure all you have to do is save that to a file called pip3, make it executable by running sudo chmod +x ./pip3, run sudo apt remove python3-pip, and then copy it to the bin directory by running sudo cp ./pip3 /usr/bin.
Here's the raw file for those who just want to download and move it.
pip3.zip

[youngxiao]

@fisadev Thanks a lot. Fix option1 really helps.

[rod-app]

@RonnyPfannschmidt

replacing the system pip using pip is always an act of system-vandalism where the one inflicting it is responsible for the fallout

is a comment of mental vandalism. As if the person doing the (naive) upgrade is intentionally setting out to damage their own installation... If that were the case, then pip itself shouldn't be nagging the user to upgrade from 9.0.1 to 10.0.1 with every single pip command executed. I myself followed that recommendation and ended up in this mess.

Fortunately:
sudo python -m pip install pip==9.0.1
was an easy enough remedy.

Blaming the victim is no answer, though.

[pradyunsg]

Hey @rod-app!

If that were the case, then pip itself shouldn't be nagging the user to upgrade from 9.0.1 to 10.0.1 with every single pip command executed.

We've noticed this and worked with OS vendors to avoid this in future versions of pip. -- #5346.

[cefn]

To address the issue I ran...

sudo geany -i /usr/bin/pip

...and edited debian's provided /usr/bin/pip to replace it with...

#!/bin/sh
# GENERATED BY CEFN
python -m pip "$@"

and the equivalent for /usr/bin/pip3 (note this invokes python3 instead).

#!/bin/sh
# GENERATED BY CEFN
python3 -m pip "$@"

...which brings back full functionality of pip despite the version 10 installed in my site-packages. I guess this will last exactly as long as debian takes to fix it, (or re-break it) by sending an updated python-pip package. Why they didn't use the package main in the first place I don't know.

Official version

The version of pip installed in .local/bin/pip shown below is a little fancier, and includes some substitution to remove -script, .py, .pyw and .exe extensions from passed arguments, but I don't know what that does or why I need it so I left it as above for simplicity.

#!/usr/bin/python

# -*- coding: utf-8 -*-
import re
import sys

from pip._internal import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

[unixnut]

I found an unrelated cause of this pip v10 issue. When upgrading pip using a very old system pip (v1.5.6 on Debian Jessie, i.e. oldstable) for which --system is the default, I noticed that incorrect scripts were installed, e.g. /usr/local/bin/pip containing from pip import main -- which I found by looking at the files. I assume this is because the older pip (or perhaps installation packages that it uses) install the .whl file incorrectly.

python -m pip install --force-reinstall pip fixed this.

[pradyunsg]

#5599 provides information and provides a single location to seek help toward resolving this issue for end users.

The comments section of that issue are open for users to discuss specific problems and solutions. :)