assertion when using --no-cache-dir in 19.0

[snstanton]

Environment

• pip version: 19.0
• Python version: 3.6.7
• OS: Linux 50de819ca3ba 4.9.125-linuxkit No instructions for how to install pip #1 SMP Fri Sep 7 08:20:28 UTC 2018 x86_64 GNU/Linux

Running in a dockerfile.

Description

The following command works with pip 18.1 and fails with 19.0.

pip3 install --no-cache-dir --upgrade -r requirements.txt

With 19.0, it fails with the following exception:

Exception:
Traceback (most recent call last):
  File "/Users/scotts/.virtualenvs/python3/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/Users/scotts/.virtualenvs/python3/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 346, in run
    session=session, autobuilding=True
  File "/Users/scotts/.virtualenvs/python3/lib/python3.6/site-packages/pip/_internal/wheel.py", line 848, in build
    assert building_is_possible
AssertionError

Removing the --no-cache-dir flag causes the install to succeed.
requirements.txt

[Mohitsharma44]

Same thing happening with:
Python v3.6.8
pip version 18.1
on
Ubuntu:latest image.

@snstanton what base image are you using? I'm seeing a similar issue on pip v18.1 as well

[gst]

I've got the exact same issue.
on my side it seems it doesn't matter which package/distribution I try to install

[dan-blanchard]

I'm seeing this even without --no-cache-dir set. It happens for all packages I try to install, even if they're already installed.

• pip version: 19.0
• Python version: 3.6.0
• OS: Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-91-generic x86_64)

I should note that in my case I'm seeing it when running pip with a combination of sudo -H and bash -l -c.

$ sudo -H bash -l -c  "/data/virtualenvs/events_beta/bin/pip install hypothesis"
Looking in indexes: https://pypi.org/simple, http://pypi.lan.cogtree.com/cogtree/simple/
Collecting hypothesis
  Downloading http://pypi.lan.cogtree.com/cogtree/simple/hypothesis/hypothesis-4.1.0-py3-none-any.whl (238kB)
    100% |████████████████████████████████| 245kB 120.5MB/s
Requirement already satisfied: attrs>=16.0.0 in /data/virtualenvs/events_beta/lib/python3.6/site-packages (from hypothesis) (18.2.0)
Exception:
Traceback (most recent call last):
  File "/data/virtualenvs/events_beta/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/data/virtualenvs/events_beta/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 346, in run
    session=session, autobuilding=True
  File "/data/virtualenvs/events_beta/lib/python3.6/site-packages/pip/_internal/wheel.py", line 848, in build
    assert building_is_possible
AssertionError

Running the same commands without -l on my bash -c, or without bash -l -c involved at all, it all works fine.

$ sudo -H bash -c  "/data/virtualenvs/events_beta/bin/pip install hypothesis"
Collecting hypothesis
  Downloading https://files.pythonhosted.org/packages/89/7b/d6206dcde963139daa03a1d85b0c3428cb3ebf2ae8de3244b14a63e22680/hypothesis-4.1.0.tar.gz (180kB)
    100% |████████████████████████████████| 184kB 33.7MB/s
Requirement already satisfied: attrs>=16.0.0 in /data/virtualenvs/events_beta/lib/python3.6/site-packages (from hypothesis) (18.2.0)
Building wheels for collected packages: hypothesis
  Building wheel for hypothesis (setup.py) ... done
  Stored in directory: /root/.cache/pip/wheels/10/12/eb/4ab734432e8466d545c8501f531458845b45e8c4427d5367f9
Successfully built hypothesis
Installing collected packages: hypothesis
Successfully installed hypothesis-4.1.0

Fascinatingly, if I run the same command without sudo or bash involved at all, it still fails with the same error, so it seems like it's some weird permissions issue maybe.

[tgs]

Another workaround for some situations

For people who hit this bug because virtualenv is automatically installing the latest version of pip, you can work around it by giving virtualenv the --no-download option, or setting VIRTUALENV_NO_DOWNLOAD=1.

But be aware that this may give you a very old version of pip, depending on the last time you upgraded virtualenv.

This also works with tox: VIRTUALENV_NO_DOWNLOAD=1 tox.

[gst]

for what it's worth : it also fails with same error if package is already installed:

gregory.starck@canon:~/tmp$ ./venv/bin/pip install --no-cache-dir six ; echo $?
Looking in indexes: http://pypi:3141/root/ax/+simple/
Requirement already satisfied: six in ./venv/lib/python3.6/site-packages (1.12.0)
Exception:
Traceback (most recent call last):
  File "/home/gregory.starck/tmp/venv/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/home/gregory.starck/tmp/venv/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 346, in run
    session=session, autobuilding=True
  File "/home/gregory.starck/tmp/venv/lib/python3.6/site-packages/pip/_internal/wheel.py", line 848, in build
    assert building_is_possible
AssertionError
2
gregory.starck@canon:~/tmp$

[cherls]

Ran into the same issue. Ended up pinning the pip version as a fix for now.

pip install --upgrade pip==18.1

[D0han]

Issue is with failing assert, so setting env PYTHONOPTIMIZE=1 (or with parameter -O) makes this error go away.
Just tested it.
This workaround works because python optimizes code removing all asserts as one of the things.
Do not go for =2 (or -OO), as this removes docstrings and other tracebacks will appear - some code wants to operate on them.

[vinnyfuria]

It looks like someone knew this might end up being an issue (source):

        # TODO: This check fails if --no-cache-dir is set. And yet we
        #       might be able to build into the ephemeral cache, surely?
        building_is_possible = self._wheel_dir or (
            autobuilding and self.wheel_cache.cache_dir
        )
        assert building_is_possible

[burkestar]

#5884 looks like this is a related change that could have caused this?

[burkestar]

Seems like the pip maintainers should rollback the recent 19 release to address this breaking change?
19.0 release notes: https://github.com/pypa/pip/blob/master/NEWS.rst#190-2019-01-22

UPDATE: not trying to cast aspersions here, was just suggesting as one way to quickly unblock people affected by this seeing as the release had just happened. Rolling forward with a hotfix works too. I appreciate the hard work of the community that supports this mission critical tooling, and agree with the sentiments below about postmortems to learn from mistakes and prevent future issues. Meanwhile, we are doing the same internally, which means a liberal amount of hardpinning pip versions in all the places :)

[cjerdonek]

The PR adding the TODO comment also has this comment in reply: https://github.com/pypa/pip/pull/5743/files#r215832743

Based on that comment and also the commenter above saying that passing PYTHONOPTIMIZE=1 makes the error go away, it seems like simply removing the assertion might be the correct fix (independent of the question of rolling back).

[tgs]

Yeah, when I delete that assert, packages do install fine with --no-cache-dir. In that case, it says it's Running setup.py install instead of Building wheel for sdist packages.

[matthewfeickert]

This is also happening to my projects. I can reproduce this in Docker images built FROM ubuntu:bionic and FROM centos:centos7 where I am installing Python 3 from source (here is a Gist that demonstrates a failing example for both of those Docker images in the event that it might be helpful). For the requirements.txt in the example in the Gist and

$ pip3 install --upgrade pip setuptools wheel
Requirement already up-to-date: pip in /usr/lib/python3.6/site-packages (19.0)
Requirement already up-to-date: setuptools in /usr/lib/python3.6/site-packages (40.6.3)
Requirement already up-to-date: wheel in /usr/lib/python3.6/site-packages (0.32.3)

then

$ pip3 install --upgrade --no-cache-dir -r requirements.txt

fails with

Exception:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/usr/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 346, in run
    session=session, autobuilding=True
  File "/usr/lib/python3.6/site-packages/pip/_internal/wheel.py", line 848, in build
    assert building_is_possible
AssertionError

but

$ pip3 install --upgrade -r requirements.txt

works fine as expected.

[asottile]

I'm particularly hitting this with tox + docker + ENV PIP_NO_CACHE_DIR=off

My workaround is to use tox-virtualenv-no-download plugin to prevent pip from auto-updating

[coderanger]

We also have --no-cache-dir in our installs inside Docker to keep the images small. Our workaround has been --cache-dir=/pipcache and then rm -rf /pipcache in the same RUN step so it never ends up in the image.

[zehauser]

Software development is hard and bugs like this are always going to happen. Certainly nobody should blame the pip maintainers or contributors for this incident.

However, I would suggest that this bug merits some sort of post-mortem analysis on the part of the pip team, due to the number of (missed) opportunities for this bug to have been caught before slipping into a general release. For example:

• automated testing of core functionality like --no-cache-dir
• pre-commit, pre-merge, or pre-release checks that flag (or prohibit) TODOs
• a (human) pre-merge review of all unresolved review comments in the PR (Github autominimizes most review comment threads when their associated code has been changed, and as of recently lets you manually mark review comment threads as resolved)
• changes in the release process — why not release a beta first and then wait several weeks before a general release?
• etc

A postmortem could result in a number of helpful betterments to ensure that software that is as core to the Python project as pip doesn't ship with bugs of this magnitude in future.

[erinxocon]

I can replicate this bug. Removing --no-cache-dir fixes it. Since I don't want it in my docker image I'm using the solution @coderanger proposed. Cheers 🌈🍰🌈

[cjerdonek]

Earlier in the thread I had asked if including --no-use-pep517 along with --no-cache-dir makes things work for people, but I didn't see a reply. For people that are still experiencing the option, can you try that?

[dmulter]

Adding the --no-use-pep517 option fixed the problem for me. Hope that helps narrow things down.

[Estartu]

pip 19.0.1 working for me in a virtualenv. But inside Jenkins (Shining Panda) it still fails. Adding --no-use-pep517 fixes the problem

[cjerdonek]

I'm reopening since some people are still experiencing the same issue.

[rifelpet]

I can also confirm that --no-use-pep517 fixed the issue for me after upgrading to pip 19.0.1 did not.

[cjerdonek]

At @pradyunsg's request, I've opened a new issue (#6197) specific to the AssertionError in the 19.0.1 release, as it's narrower in scope and will need new investigation. So I'm reclosing this issue.

[marcstreeter]

Ran into the same issue. Ended up pinning the pip version as a fix for now.

pip install --upgrade pip==18.1

or your FROM python:3.6-alpine can change to FROM python:3.6.7-alpine

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.