Add option to use wheel when installing packages

[pytoxbot]

• Bitbucket: https://bitbucket.org/hpk42/tox/issue/106
• Originally reported by: @vmalloc
• Originally created at: 2013-07-24T14:46:03.442

The new pip 1.4 (and virtualenv 1.10) support the wheel package format, and can automatically cache downloaded and built packages in a custom location. This would be extremely useful for tox, as rerunning it multiple times will require no extra build time...

[pytoxbot]

Original comment by peteut

Something like a pre_deps would be helpful, so wheel would be in place when pip installs deps.

[testenv]
pre_deps =
    pip>=7.0.0
    wheel
deps = -rrequirements.txt

[pytoxbot]

Original comment by marcelm

You can solve the problem in two ways; both make sure that pip 7.0 or later is installed in the virtualenvs that tox creates before it runs setup.py within them.

The first option is to install tox into a virtual environment and run it from there. The installation will pull in a recent version of virtualenv, which in turn comes with a recent version of pip.

The second option is to add pip and wheel to the [testenv] section in tox.ini:

[testenv]
deps =
    pip>=7.0.0
    wheel

All packages listed there will be installed with the old pip version, but then the setup.py is run with the new one.

It seems to me that no changes to tox itself are really required (anymore). As soon as the distributions come with more recent virtualenv versions, this will have solved itself. And if you don’t use a distribution package, then it already works!

[pytoxbot]

Original comment by @RonnyPfannschmidt

as far as i understood it would be suficient if to would install a cached suffuciently recent pip version, pip 7.0 will go as far as building and caching wheels locally of everything

[pytoxbot]

Original comment by @blueyed

wheelhouse=yes would be good, and could be even used by default.

However, pip might do this automatically in the future (creating wheels in a cache and installing it from there).
But that's against the idea of isolation then again.

[pytoxbot]

Original comment by @hpk42

What about introducing a wheelhouse=yes and then configure the pip commands accordingly? The pip configuration would then use a per-tox dir configuration, not a per-user one. I am slightly skeptical about using per-user directories as this violates tox-run isolation.

[pytoxbot]

Original comment by @blueyed

Just for information and reference: you can make tox use wheels by configuring
pip globally (for the user):

[global]
download_cache = ~/.cache/pip/download
wheel_dir = ~/.cache/pip/wheelhouse
find_links =
    /home/user/.cache/pip/wheelhouse
    /home/user/.cache/pip/download

And to create the wheels:

.tox/py27/bin/pip install wheel
.tox/py27/bin/pip wheel -r test_requirements.txt

[pytoxbot]

Original comment by @hpk42

Actually, supporting wheel house should remain its own issue as it's not clear that issue35 will fully address this. @vmalloc and others: could you check how much PR61's "installer_command" support would help with implementing wheel-house support?

[pytoxbot]

Original comment by @hpk42

Duplicate of #35.

[pytoxbot]

Original comment by @vmalloc

+1

[pytoxbot]

Original comment by @bhodorog

I do think it would be useful to extend the proposed behavior by having a option (globally and maybe also override-able for each environment) which would allow the user to specify it's own options to the "core" pip install command which installs the project's sdist.

I've just been hit by the need to have the package installed with pip1.4 with --pre (pre-development releases). Since others have similar needs (--wheel, I suspect other options to pip install would be usefull) I guess having an option in the ini file like:

pip_install_options="any valid option specified by pip install --help command"

[pytoxbot]

Original comment by @hpk42

Needs some experimentation but sounds interesting. Any chance that you could tweak tox yourself and see how well it works? There already is a mechanism for detecting if an venv needs to be re-created, so i guess it's not too hard to try adding the options, and then to try it out on some real projects.

[pytoxbot]

Original comment by @vmalloc

I believe the greatest benefit would come if running "pip wheel --wheel-dir=X --use-wheel ." on the current project dir.

This would package the current module, as well as any dependency, to the specified "X" directory (~/.toxcache, or a similar name). Then the installation of the package in the virtualenv can be done with "--use-wheel --find-links=X", which can speed up installation significantly. A good example for this that I've run into is packages depending on lxml, pyzmq or sqlalchemy, requiring some heavy compilation which takes a lot of time and CPU... Wheel speeds up these cases to mere seconds.

[pytoxbot]

Original comment by @hpk42

What would be involved exactly on tox side?

[obestwalter]

@vmalloc, @blueyed dredging this up to see what to do about this. Would you like to follow up on this or has the passing of time and versions of tox and other tools made this obsolete?

[RonnyPfannschmidt]

im tempted to just declare that recent dev/ci envs should not have pip versions that cant handle the automtic wheel caching

[obestwalter]

@RonnyPfannschmidt I would tend to think the same :)

[vmalloc]

Sorry for chiming in so late, didn't see the mention previously. I'm only using recent pip versions so the issue is solved for me. I'm fine with relying on the automatic wheel caching logic