Support built-in “venv”

[uranusjr]

(Try to) fix #67.

Essentially this only adds a new way to create venvs, that calls python -m venv instead of python -m virtualenv. It is used by default, unless either 1. venv is not available, or 2. the PEW_NO_BUILTIN_VENV PEW_USE_VIRTUALENV environment variable is not empty.

I played around this a little and everything seems to work; the two modules are just too similar. I still pull the logic into their own classes, just in case something acts differently and requires additional abstraction.

Some others things that might need considering:

1. venv is completely silent when creating a virtualenv. virtualenv is pretty verbose in comparison. Maybe it is better to add some additional logging?
2. As mentioned in Start supporting pyvenv #67, ensurepip still has some issues. While the Debian problem is largely gone, the --system-site-package one is not. It is possible to patch this manually in pew, but is it worth it? I’ve decided to ignore that. Just use virtualenv if ensurepip is not available.
3. There is no known virtualenv-clone equivalent for venv, so there’s no pew cp for venv. I’ve skipped related tests (all failing), but an alternative is needed. It seems to work perfectly.

[uranusjr]

Oh man so sitepackages_dir and wipeenv_cmd, totally working on my machine, are breaking the tests. This is gonna be difficult.

[uranusjr]

Finally, only one to go! test_wipe really stumps me though… I can reproduce it locally with Tox, but the test passes if I run it globally. I don’t understand.

[uranusjr]

I think I’ve got it. Tests on Appveyor and Travis are run inside a virtualenv (through Tox in Appveyor’s case), and virtualenv does not create entry points if you create a venv inside it. To reproduce:

$ which python3.6
/usr/local/bin/python3.6
$ python3.6 -m venv env1
$ ls env1/bin
activate  activate.csh  activate.fish  easy_install   easy_install-3.6   pip  pip3  pip3.6  python   python3
$ python3.6 -m virtualenv env2
...
$ env2/bin/python -m venv env3
$ ls env3/bin
activate  activate.csh  activate.fish  python  python3

You can see pip entry points are missing. I’m not sure what is the problem here, and if there is a workaround to it, but this is really a show-stopper for venv. I’ll have to leave this here until someone can provide more information on how this can be resolved.

[uranusjr]

So it seems the nested virtual environment thing is a known problem, and I’ve found enough information on BPO to solve it. Now if only I can make the Windows build pass…

https://bugs.python.org/issue30811

[uranusjr]

So it is impossible for venv to work properly on Windows. I’m starting to wonder if venv is really useable enough to be included.

[uranusjr]

Alright so I think this is ready for review. In summary, this patch adds support for creating and managing virtual environments with the built-in venv module. This is available for Python 3.4+ on non-Windows platforms.

• Windows is excluded because Pip usage in a venv is broken if Pew is installed inside a virtualenv, and used to create a venv. https://bugs.python.org/issue30811
• Python < 3.3 are excluded for lack of venv.
• Python 3.3 is excluded because it lacks ensurepip.

venv usage is enabled by default if the system meets the above requirements. Users can opt out (and switch back to the virtualenv backend) by setting PEW_USE_VIRTUALENV to a non-empty value.

A _venv module is added to provide plug-n-play support of both virtual environment backends. The main CLI uses choose_backend() to choose a backend based on the surrounding system, and guess_backend() to decide what backend should be used for an existing virtual environment. Certain operations are refactored into _venv.Backend subclasses to provide different behaviours. A _cfg module is also added to support parsing of pyvenv.cfg. This file stores information about a venv.

Tests are mostly unchanged, but I cleaned up some of them to make my debugging easier, and I think they are beneficial to the codebase. A few tests are marked as skipped because they rely to much on virtualenv internals, and I feel it is too difficult to rewrite them to work with both virtualenv and venv.

I have no idea why the Nix build is failing. I didn’t change anything about it (I think), and lack knowledge on its internals to know why it breaks.

[FranklinYu]

So Windows is broken only with venv nested inside virtualenv? Not when using venv alone?

I’m not sure whether venv is designed to be nested. For example virtualenv can’t be nested. Also see this Python issue.

[uranusjr]

virtualenv can’t handle nesting either, but is smart enough to find the “real” Python when creating the inner virtualenv, avoiding nesting altogether:

C:\Users\uranusjr\Desktop>python -m virtualenv outer
Using base prefix 'C:\\Users\\uranusjr\\AppData\\Local\\Programs\\Python\\Python36'
New python executable in C:\Users\uranusjr\Desktop\outer\Scripts\python.exe
Installing setuptools, pip, wheel...done.

C:\Users\uranusjr\Desktop>outer\Scripts\pip install -q virtualenv

C:\Users\uranusjr\Desktop>outer\Scripts\python -m virtualenv inner
Using real prefix 'C:\\Users\\uranusjr\\AppData\\Local\\Programs\\Python\\Python36'
New python executable in C:\Users\uranusjr\Desktop\inner\Scripts\python.exe
Installing setuptools, pip, wheel...done.

You can see from the last command’s output, it uses the original Python prefix, not the virtualenv one.

venv, on the other hand, does not do this “real Python” lookup, and would happily use the inner Python, causing the issue.

---

But this reminds me, how does virtualenv do this anyway? I haven’t thought about this. Maybe I should go check its source and see if it can provide a workaround.

[uranusjr]

Aha! virtualenv patches sys to set a real_prefix attribute, so it can tell if it is inside a virtualenv. This should provide a good enough workaround. Let me try it out.

• Related changelog
• Where this is done

---

Update:

Alas, real_prefix maps to prefix, which is for platform-independent files. The binary should use exec_prefix, which is not stored by virtualenv. This won’t work. (Actually it usually would, but won’t for certain edge cases. I want a better solution.)

[FranklinYu]

I was thinking about the possibility to disable virtualenv in Travis CI or AppVeyor? After all, nested venv is not a typical use case…

[uranusjr]

So I implemented a way to “guess” where python.exe is. This will work if you have a reasonable installation, i.e. have sys.base_prefix and sys.base_exec_prefix set to the same location, and havepython.exe available in a popular subdirectory (in the top level, inside Scripts or bin, etc.). This should work most of the time, and when it doesn’t we’ll raise an error.

[FranklinYu]

The only failing test is in Nix, where system denied access to /tmp/nix-build-python3.6-pew.drv-0/WORKON_HOME/destination/bin/activate.csh? Not familiar with Nix, but shouldn’t /tmp be accessible to everyone? Or it means that the path is not available?

[berdario]

I think we should make a Path out of bindir straight away

[berdario]

Wow, thanks for the all the work on this.

Maybe another solution could be to ask to always provide the --python option when using the builtin venv? (this would be an obstacle for opting in people automatically, though)

edit: Nevermind, I see that you already propose it to the user when failing

There's only a bit that I wouldn't approve straight away and that I'll have to bikeshed a bit: the Backend class seems a bit an overkill, given that all the methods in it and the subclass are basically static (there won't be more than 1 backend instantiated at the same time)... it should be able to be easily rewritten with a NamedTuple and plain functions (but I can do that at merge time)

Anyhow, I'll have a quick look at the Nix failure soon

[berdario]

Ok, the Nix failure is one of those:

why does this not work?
...1 hour later...
how could have this ever worked?

When venv creates a virtualenv, all of the activate scripts have permission 444 (that is, read only for everyone). clonevirtualenv subsequently does a shutil.copytree(, which preserves the permission, and thus it fails when touching the scripts.

The old virtualenv tool instead creates the permissions as 644, allowing the user to modify it. (I think the 444 is cleaner, since those scripts are never supposed to be modified)

I also reproduced this manually on Macos, by creating a venv in /tmp, and then invoking clonevirtualenv.

This was a bit frustrating to debug, since running instead the test inside Nix locally didn't trigger this (somehow, the permissions for the activate scripts are 644 just like for plain virtualenv). Also, since High Sierra the test collection of py.test seem to have slow down incredibly :/

Anyhow, this isn't a spurious error due to environment/configuration, so it should be fixed... either upstream in virtualenv-clone, or by ensuring that such activate scripts have write permission before attempting a copy of the virtualenv.

[berdario]

I'd place this in pew._utils, and reuse it in pew._venv

[uranusjr]

Wow, that is intense debugging. Well done! Since the pew cp tests are also working, I’m guessing this probably has something to do with the venv being created in /tmp, a directory not owned by the current user.

Anyway, I agree this should be fixed. I’ll probably work on this (and other minor issues you mentioned) this weekend. Thanks for the effort and review!

[berdario]

Actually, it's simpler than it relying on /tmp: the root cause is all here

The mode is copied from the source file, so if the source activate.csh has been installed with read-only permissions, the virtualenv scripts will also be installed with read-only permissions. I don't think it's a good idea for venv to do this.

[kennethreitz]

this is super problematic, in my experimentations. python -m venv doesn't install pip, for example, on ubuntu systems.

[cournape]

Another argument to use venv: it seems to fix the issue where using pew on OS X breaks packages that need "framework builds", like matplotlib + MacOSX backend

[asottile]

Just to add some more links to this issue, this appears to be the pypa/virtualenv related issue: pypa/virtualenv#1095

[DougPlumley]

I went down the rabbit hole of trying to get pipenv working with venv so it wouldn't break autocomplete in my Python interpreter...I ended up here.

Trying to further my understanding of Python, development workflow, etc. has there been any progress in this space or should I read up more and try to make a meaningful contribution?

[FranklinYu]

@DougPlumley No progress. Contribution welcome of course.