virtualenv fails when run from a venv

[pfmoore]

Environment:
Python 3.7.3, 64 bit, on Windows 7. Note this does not happen with Python 3.6.8, it seems to be specific to Python 3.7.

Steps to reproduce:

py -m venv xxxx
xxx\scripts\python -m pip install virtualenv
xxx\scripts\virtualenv.exe yyy

Possible diagnosis:
Virtualenv and venv use different means of locating the "base" Python interpreter (sys.real_prefix and sys.base_prefix). It's possible that these are clashing somehow.

It's also possible that some of the work around https://bugs.python.org/issue35872 might be relevant.

Impact:
This means that tools that manage virtualenvs, such as nox or tox, cannot be run from a Python 3 venv. The pipx tool does precisely this, and so pipx cannot be used to manage those tools if used in this environment.

Test case output:

>py -m venv xxxx
>xxxx\scripts\python -m pip install virtualenv
Collecting virtualenv
  Using cached https://files.pythonhosted.org/packages/33/5d/314c760d4204f64e4a968275182b7751bd5c3249094757b39ba987dcfb5a/virtualenv-16.4.3-py2.py3-none-any.whl
Installing collected packages: virtualenv
Successfully installed virtualenv-16.4.3
>.\xxxx\scripts\virtualenv.exe yyyy
Using base prefix 'C:\\Users\\pfmoore\\AppData\\Local\\Programs\\Python\\Python37'
New python executable in C:\Work\Scratch\n\yyyy\Scripts\python.exe
No pyvenv.cfg file
ERROR: The executable C:\Work\Scratch\n\yyyy\Scripts\python.exe is not functioning
ERROR: It thinks sys.prefix is 'c:\\work\\scratch\\n' (should be 'c:\\work\\scratch\\n\\yyyy')
ERROR: virtualenv is not compatible with this system or executable
Note: some Windows users have reported this error when they installed Python for "Only this user" or have multiple versions of Python installed. Copying the appropriate PythonXX.dll to the virtualenv Scripts/ directory may fix this problem.

[pfmoore]

Further diagnosis - I'm not sure this is about base_prefix. It looks like Python changed the specific executable that's copied into a venv, so it's no longer the same as the executable in the base interpreter, and it appears to be what's producing the "No pyvenv.cfg file" message.

There's a note about this in the What's New document saying this was introduced in 3.7.2, but I can't find anything that gives any details that would explain why it's broken virtualenv.

I believe this change was introduced by @zooba - maybe he can offer some insight?

By the way, I'm not sure this counts as an "enhancement" - it's a fairly straightforward breakage (admittedly of a relatively rare case), something that works in 3.6 has broken in 3.7, so even though it was broken by changes in Python 3.7, surely this would be covered by support of 3.7? Not that it matters much how we label it, what's important is fixing it.

I tried to develop a fix by getting the actual executable from sys.base_prefix, rather than getting the redirector, but that didn't work. Again, maybe @zooba could help here - I'd be happy to work with him on fixing this, as I believe it's going to keep hitting people otherwise.

[zooba]

You may need to use the semi-secret sys._base_executable, rather than base_prefix. This will be available on Windows and macOS in a venv. Creating a venv from within a venv is apparently not supported on Linux.

Last time I looked at virtualenv it still wasn't using venv internally - has that changed?

[pfmoore]

Creating a venv from within a venv is apparently not supported on Linux.

Really? Do you know why, or have any links with more information? That has the potential to break things like pipx, which run commands from a virtualenv (and I think it uses venv in recent versions). If you install a tool like tox (or virtualenv itself) that creates virtual environments, things are going to go wrong. As far as I know, virtualenv handles creating an environment from another one OK, so this would be a non-trivial limitation of venv, if it's the case.

Last time I looked at virtualenv it still wasn't using venv internally - has that changed?

Not yet, but see #1343. That might be a non-starter, though, if the above limitation of venv is true :-(

[zooba]

See bpo35978, but on re-reading it looks like I inferred from the test skips that venv-from-within-venv wasn't supported.

It shouldn't be difficult to make it work on Linux - here is the line in venv that would handle it and this is the section in site that sets sys._base_executable on macOS and Windows. I'm not sure exactly how to set it for Linux, but I suspect it's not in that file. (And of course, a venv is currently just broken if you pass -S to avoid import site, so venv-from-venv probably doesn't work there either.)

I don't recall all the details now, but I think the problem is that venv in a venv puts the wrong home directory in the config unless it actually locates its base runtime, which it needs the base sys.executable for and not just sys.base_prefix.

[pfmoore]

Thanks. Given that it's already broken in existing Python versions, virtualenv will need to work with the problem, so for now I think I'll concentrate on workarounds rather than fixing core python.

I am a bit worried if venv still has as many rough edges as you suggest - it's going to be hard for virtualenv to switch to use venv behind the scenes if that would result in breakages, but on the other hand if we don't switch, how will any bugs in venv ever get enough exposure to get them fixed? I'd assumed that since venv was released in 3.3 or so, it would be reasonably solid by now - you've made me worried about that assumption, though :-(

[zooba]

I don't think you could ever create a venv from within a venv, there certainly wasn't anything in the code to make it work and the tests were all skipped in this case.

Ignoring site would break virtualenv too, last I checked. -S isn't really a useful option unless you know what you're doing, and certainly not if you've been following the recommended approaches to using Python these days.

But the core approach to resolving the base environment from the symlink or equivalent is likely more solid in venv than virtualenv, if only because it's tied to the exact version of Python you're using.

(But yeah, still flimsy in a lot of ways. I tried to come up with a way to avoid all this mess...)

[pfmoore]

Just adding a quick note here - in spite of #1345, I suspect there may still be some things going wrong. I encountered some weird errors in the pipx test suite (pipx uses venv, and tox uses virtualenv, so we have venvs created from virtualenvs, although the tox-venv plugin is used, so maybe we have venvs created from venvs).

I don't really know (yet) what's going wrong there, but the whole situation is a frustrating mess.

[gaborbernat]

Slightly related to this but somehow moving the parser from optparse to argparse (see #1353) started breaking the venv test on python 3.7.2 😆 Haven't figured it out yet, why or how.

[pfmoore]

If I click on "Details" against the build failure in that PR, I get "build not found" :-( If I can get a look at the failure, I'll see if I can see what's up.

[pradyunsg]

somehow moving the parser from optparse to argparse (see #1353) started breaking the venv test on python 3.7.2

Gee. Don't scare me about pypa/pip#4659.

[gaborbernat]

@pradyunsg false positive 😄 all is good now; then again why would you want click when the built-in argparse is much more flexible and built-in (so no need to vendor it). I'm probably alone in not liking how click looks and behave at a cost of extra dependency.

[pradyunsg]

Oh, because of one quirky bug in argparse on older versions of Python (there's a comment listing that on that issue).

Plus, I don't plan to use the decorator API of click (I really strongly dislike that approach) but instead the internal (public, documented) classes that are really just a nicer API than argparse.

[pfmoore]

Just adding a quick note here - in spite of #1345, I suspect there may still be some things going wrong.

Found it. The code in #1345 does not take account of the possibility that someone could run

    py -m venv xxx
    full\path\to\xxx\scripts\python.exe virtualenv -p full\path\to\xxx\scripts\python.exe

i.e., using the -p flag to explicitly point to the Python interpreter installed into a venv (which isn't a full interpreter but the redirector/launcher).

I don't yet know how to fix this case, but I'll look into it.

[gaborbernat]

@pfmoore FYI the master is now broke, seems Azure upgraded to 3.7.3 and that broke somehow the CI for this test.

[pfmoore]

Huh? 3.7.3 shouldn't be broken, it's what I use on all my PCs. Unless it's because the usage in tox is broken (which is this issue I flagged just above). I have a probably-OK fix for that, but it restructures the logic in my original patch a bit and I'd prefer to check it a bit further before saying it's OK. I might put a PR up though to see if it fixes the CI.

[gaborbernat]

@pfmoore can we close this then now?

[pfmoore]

Yeah, I'm pretty sure we've covered all the corner cases now (and if we haven't I can open a new issue).

[pradyunsg]

Thanks for tackling this @pfmoore and @gaborbernat!