bootstrap -> No module named pip

[jedie]

I created a minimal bootstrap with virtualenv v1.11.6
A run results in this:

$ python2 test-bootstrap.py ~/test_py2
extend_parser called.
adjust_options args: ['/home/jens/test_py2']
New python executable in /home/jens/test_py2/bin/python2
Not overwriting existing python script /home/jens/test_py2/bin/python (you must use /home/jens/test_py2/bin/python2)
Cannot find a wheel for setuptools
Cannot find a wheel for pip
Installing setuptools, pip...
  Complete output from command /home/jens/test_py2/bin/python2 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip:
  Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named pip
----------------------------------------
...Installing setuptools, pip...done.
Traceback (most recent call last):
  File "test-bootstrap.py", line 2355, in <module>
    main()
  File "test-bootstrap.py", line 826, in main
    symlink=options.symlink)
  File "test-bootstrap.py", line 994, in create_environment
    install_wheel(to_install, py_executable, search_dirs)
  File "test-bootstrap.py", line 962, in install_wheel
    'PIP_NO_INDEX': '1'
  File "test-bootstrap.py", line 904, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /home/jens/test_py2/bin/python2 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip failed with error code 1

All files are here: https://gist.github.com/jedie/4bc671239b8959376d4b

more info:

$ python2 --version
Python 2.7.6
$ python2 -c "import pip;print pip.__version__"
1.5.4

$ /home/jens/test_py2/bin/python --version
Python 2.7.6
$ /home/jens/test_py2/bin/python -c "import pip;print pip.__version__"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named pip

Maybe it's related to pypa/pip#1918 ?

[jedie]

btw. it works with python3:

$ python3 test-bootstrap.py ~/test_py3
extend_parser called.
adjust_options args: ['/home/jens/test_py3']
Using base prefix '/usr'
New python executable in /home/jens/test_py3/bin/python3
Also creating executable in /home/jens/test_py3/bin/python
Installing setuptools, pip...done.
after_install from '/home/jens/test_py3'

Some more info:

$ python3 -V
Python 3.4.0
$ python3 -m pip -V
pip 1.5.4 from /usr/lib/python3/dist-packages (python 3.4)

$ ~/test_py3/bin/python -V
Python 3.4.0
$ ~/test_py3/bin/python -m pip -V
pip 1.5.6 from /home/jens/test_py3/lib/python3.4/site-packages (python 3.4)

[jedie]

btw. i have python-pip and python-pip3 installed. Both are: 1.5.4-1 and the package python-virtualenv 1.11.4-1 was installed.

I have removed all pip/virtualenv packages and re-try again my test-bootstrap.

Now it will fail with python2 and python3 with the same error.

[jedie]

I created a fresh new venv with python3 and installed pip and virtualenv from github:

(test_venv) ~/test_venv $ bin/python -m virtualenv --version
12.0.dev1
(test_venv)  ~/test_venv $ bin/python -m pip -V
pip 6.0.dev1 from /home/jens/test_venv/lib/python3.4/site-packages (python 3.4)

...and recreate the bootstrap script and try it: Same "No module named pip" error.

[pfmoore]

Looks like your bootstrap script doesn't have the necessary virtualenv_support directory alongside it.

[jedie]

Can you explain a little bit more, what you mean?

My script is not really more, than from the docu here: https://virtualenv.pypa.io/en/latest/virtualenv.html#bootstrap-example

[pfmoore]

When you create a bootstrap script, it is essentially just a customised version of virtualenv.py. But as noted in https://virtualenv.pypa.io/en/latest/virtualenv.html (under "Installation", at the end of that section, the blue note), any virtualenv.py script (vanilla or customised via the bootstrap process) must " include a virtualenv_support directory alongside virtualenv.py which contains a complete set of pip and setuptools distributions".

So, wherever your bootstrap script is located, you need to create a directory called virtualenv_support next to it, and put wheels for pip and setuptools in there. These can either be from PyPI, or you can get the ones from your virtualenv installation.

[jedie]

Ah! Thanks for the Information!

That's really boring. So a single bootstrap file is a little bit useless, if other files must be downloaded before :(

Was that always been so?

EDIT: Seems that #632 is a similar request for a self contained virtualenv.py...

btw. i think the error message is not really good. Maybe a simple solution like the following is more helpfull, e.g.: :

try:
    import pip
except ImportError as err:
    sys.stderr.write("Please install pip fist, before bootstrap! (Origin error: %s)" % err)
    sys.exit(-1)

EDIT2: The "node: pip is needed" should be added in the https://virtualenv.pypa.io/en/latest/virtualenv.html#creating-your-own-bootstrap-scripts section, too.

[pfmoore]

It's been that way for a long while. virtualenv used to download pip and setuptools from the internet, but that was potentially insecure, often very slow, and unreliable (your internet goes down and you can't create virtualenvs). The change was deliberate, and while it does remove the "single file application" option, we viewed it as worth the cost.

[jedie]

What's about to include pip into virtualenv.py?

Or maybe better a separate "bootstrap" project, that's put everything together in one file?

[pfmoore]

I've been through this thought process a long time ago :-)

We use the pip wheel to install itself and setuptools. To do that we add the pip wheel to sys.path, using the fact that wheel files are actually zip files that can be added to sys.path (with some limitations, that don't apply here). Embedding the pip wheel into virtualenv wouldn't allow that.

Sorry...

[jedie]

One Question: It also doesn't work if pip is systemwide installed, isn't it?

On creating i see this:

Cannot find sdist setuptools-*.tar.gz
Cannot find sdist pip-*.tar.gz

$ /home/jens/testenv/bin/python -m pip -V
/home/jens/testenv/bin/python: No module named pip

So my added info ( jedie/PyLucid@dd3b5db ) is simply wrong :(

I really must have setuptools and pip as sdist ?

The a installation will result in this:

1. create a virtualenv_support directory
2. download setuptools sdist file and store it in virtualenv_support
3. download pip sdist file and store it in virtualenv_support
4. download bootstrap file
5. run bootstrap

That much to do, isn't it?

One reason to remove the functionality was that it's potentially insecure...
But now, the user must download setuptools and pip and has to check the integrity by hand, isn't it?
So, he must do this:

1. create a virtualenv_support directory
2. download setuptools sdist file and store it in virtualenv_support
3. verify download
4. download pip sdist file and store it in virtualenv_support
5. verify download
6. download bootstrap file
7. verify download
8. run bootstrap

I'm wrong?

[pfmoore]

setuptools and pip as wheels not sdists, but yes.

You manually download over https (which is secure). The previous implementation used http (which is not).

And why not just use the virtualenv_support that is installed with your virtualenv distribution? Surely that's easier than downloading, if that's what matters to you?

[jedie]

You manually download over https (which is secure). The previous implementation used http (which is not).

http is not secure, yes. But IMHO a better solution would be to add hashes and compare that, like the solution from #632 (comment) then it's euqal if https or http is used. (https is also not secure, if the server was hacked and the files are replaced ;) )

And why not just use the virtualenv_support that is installed with your virtualenv distribution? Surely that's easier than downloading, if that's what matters to you?

Seems that i understand this not right. There is a way to skip the wheel downloads?
I search for "virtualenv_support" in the docs: https://virtualenv.pypa.io/en/latest/search.html?q=virtualenv_support&check_keywords=yes&area=default

It's only a simply name of a directory, which can contain the wheels... So how can this help here?!?!

Yes, i can bundle my bootstrap + the virtualenv_support directory with the wheel files in a archive... Is that what you mean?

[pfmoore]

What are you trying to do? The following works:

bootstrap.py
virtualenv_support
        pip-1.5.6-py2.py3-none-any.whl
        setuptools-3.6-py2.py3-none-any.whl

You don't put them in an archive, you have them as separate files. There is no single-file solution, if that's what you want, sorry.

[jedie]

The following works:
...

Yes, i understand that... But my usecase is this:

A user would like to install PyLucid CMS. In the past he must only do that:

$ wget https://raw.github.com/jedie/PyLucid/master/bootstrap/pylucid-boot.py
$ python pylucid-boot.py PyLucid_env 

see: http://www.pylucid.org/permalink/333/1-create-a-pylucid-environment-with-pylucid-boot#bootstrapping

This doesn't work since virtualenv v1.10 :(

Now the user must also have pip-1.5.6-py2.py3-none-any.whl and setuptools-3.6-py2.py3-none-any.whl and put theses files in virtualenv_support...

This is far away from, just "wget and run" :(

There is no single-file solution, if that's what you want, sorry.

Yes that is what i want. I requested this here: pypa/packaging-problems#55

Maybe i will include the complete "get_pip.py" in my bootstrap... i'm working on a proof of concept...

[techtonik]

Run into the same problem. =/ Looks complicated and overengineered/overhacked.

[techtonik]

I am ready to deploy my own installer if that stuff - https://bitbucket.org/pypa/wheel/issue/111/how-to-install-wheel-manually - is clarified.

[techtonik]

This exception definitely need a better error message.

[jedie]

My solution is https://github.com/jedie/bootstrap_env to create a complete self contained virtualenv bootstrap file by enbed 'get-pip.py' ;)

But yes, a better error message would be good.

[Ivoz]

BTW you can create a virtualenv WITHOUT pip or setuptools, in which case virtualenv won't complain about them not being around. Use the --no-setuptools flag. But a virtualenv without setuptools or pip tends not to be very useful for most usecases.

But in practice you should be able to run this, then run a local get-pip.py with the virtualenv activated, and I think that would work... haven't tested personally though.

You also don't specifically need a directory called virtualenv_support, you can list another directory with the --extra-search-dir= flag.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Just add a comment if you want to keep it open. Thank you for your contributions.