python -m venv from inside pipenv shell

[dimaqq]

Hi Pipenv devs!
[obligatory nod for creating an awesome tool]

We have a workflow where a virtual env is created on the fly for some hairy tests.

There's a difference between running python -m venv from under pipenv and global interpreter:

> ls -lt venv-from-pipenv/bin/
total 24
-rw-rw-r--  1 xxx staff  2211 Dec  4 16:08 activate
-rw-rw-r--  1 xxx staff  1267 Dec  4 16:08 activate.csh
-rw-rw-r--  1 xxx staff  2419 Dec  4 16:08 activate.fish
lrwxr-xr-x  1 xxx staff     6 Dec  4 16:08 python3 -> python
lrwxr-xr-x  1 xxx staff    72 Dec  4 16:08 python -> /Users/xxx/.local/share/virtualenvs/xxx-x/bin/python

> ls -lt venv-from-system-python/bin/
total 64
-rw-rw-r--  1 xxx staff  2211 Dec  4 16:07 activate
-rw-rw-r--  1 xxx staff  1267 Dec  4 16:07 activate.csh
-rw-rw-r--  1 xxx staff  2419 Dec  4 16:07 activate.fish
-rwxr-xr-x  1 xxx staff   247 Dec  4 16:07 pip3.7
-rwxr-xr-x  1 xxx staff   247 Dec  4 16:07 pip3
-rwxr-xr-x  1 xxx staff   247 Dec  4 16:07 pip
-rwxr-xr-x  1 xxx staff   265 Dec  4 16:07 easy_install-3.7
-rwxr-xr-x  1 xxx staff   265 Dec  4 16:07 easy_install
lrwxr-xr-x  1 xxx staff     9 Dec  4 16:07 python3 -> python3.7
lrwxr-xr-x  1 xxx staff     9 Dec  4 16:07 python -> python3.7
lrwxr-xr-x  1 xxx staff    24 Dec  4 16:07 python3.7 -> /usr/local/bin/python3.7

pip is missing in the first venv.
[in the 2nd venv, it works correctly]

Running venv/bin/python -m pip in the first venv reads|modifies "parent" (pipenv's) site-packages, not the new venv's.
[in the 2nd venv, it works correctly]

Is this workflow supported?
Am I doing something completely wrong?
Any other comments?

pipenv, version 2018.10.13

[frostming]

Can you please upgrade pipenv and see if the issue still exists?

From your description, the replicating steps:

• $ pipenv run python -m venv venv-from-pipenv
• $ python -m venv venv-from-system-python

[dimaqq]

Yes the issue still persists.

Thanks for the simpler steps to reproduce :)

~> pipenv run python -m venv venv-from-pipenv
Creating a virtualenv for this project…
Pipfile: /Users/xxx/Pipfile  # [there was none, file was created]
Using /usr/local/Cellar/pipenv/2018.11.26/libexec/bin/python3.7 (3.7.1) to create virtualenv…
⠧ Creating virtual environment...Already using interpreter /usr/local/Cellar/pipenv/2018.11.26/libexec/bin/python3.7
Using real prefix '/usr/local/Cellar/python/3.7.1/Frameworks/Python.framework/Versions/3.7'
New python executable in /Users/xxx/.local/share/virtualenvs/xxx-x/bin/python3.7
Also creating executable in /Users/xxx/.local/share/virtualenvs/xxx-x/bin/python
Installing setuptools, pip, wheel...
done.
✔ Successfully created virtual environment!

~> venv-from-pipenv/bin/python -m pip install requests
[skipping most output]
Successfully installed certifi-2018.11.29 chardet-3.0.4 idna-2.7 requests-2.20.1 urllib3-1.24.1

~> pipenv run python -m pip freeze
certifi==2018.11.29
chardet==3.0.4
idna==2.7
requests==2.20.1
urllib3==1.24.1

Et voila, packages got installed in pipenv's own venv, not the target venv.

$ pipenv --support

Pipenv version: '2018.11.26'

Pipenv location: '/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv'

Python location: '/usr/local/Cellar/pipenv/2018.11.26/libexec/bin/python3.7'

Python installations found:

• 3.7.1: /usr/local/bin/python3
• 3.7.1: /usr/local/bin/python3.7m
• 2.7.10: /usr/bin/python
• 2.7.10: /usr/bin/pythonw
• 2.7.10: /usr/bin/python2.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.1',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.7.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.7.0: Wed Oct 10 23:06:14 PDT '
                     '2018; root:xnu-4570.71.13~1/RELEASE_X86_64',
 'python_full_version': '3.7.1',
 'python_version': '3.7',
 'sys_platform': 'darwin'}

System environment variables:

• PATH
• TERM_PROGRAM
• TERM
• SHELL
• TMPDIR
• Apple_PubSub_Socket_Render
• TERM_PROGRAM_VERSION
• TERM_SESSION_ID
• USER
• COMMAND_MODE
• SSH_AUTH_SOCK
• __CF_USER_TEXT_ENCODING
• PWD
• EDITOR
• ITERM_PROFILE
• XPC_FLAGS
• XPC_SERVICE_NAME
• SHLVL
• HOME
• COLORFGBG
• ITERM_SESSION_ID
• LOGNAME
• LC_CTYPE
• SECURITYSESSIONID
• COLORTERM
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_SHIMS_BASE_MODULE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /usr/local/Cellar/pipenv/2018.11.26/libexec/tools:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin
• SHELL: /usr/local/bin/fish
• EDITOR: vim
• PWD: /Users/xxx

---

Contents of Pipfile ('/Users/xxx/Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]

[requires]
python_version = "3.7"

[frostming]

@dimaqq Thanks for your reply, I found venv doesn't work well with virtualenv. I recommend you file an issue to virtualenv, or perhaps python venv.

$ virtualenv venv
$ venv/bin/python -m venv venv-from-virtualenv
$ venv-from-virtualenv/bin/python -m pip --version
# The pip points to the venv/bin/pip

[techalchemy]

There is a branch with experimental venv support and @uranusjr is the expert on that. It’s not officially supported but the correct workflow is to make the venv yourself, since pipenv will always make a virtualenv as step 1 in the process of running any command which means you’re basically attempting to make a virtualenv, use that interpreter to make a venv.

Note that venv and virtualenv are both different and incompatible in important ways, and specifically virtualenv does not support venv because venv is python 3 only while virtualenv is python2 compatible. Pipenv generates environments with virtualenv and I doubt you’ll be able to make venvs from there. Most likely you can make a script that does something like eval $(python -c 'import sys; print(os.path.join(getattr(sys, "real_prefix", sys.prefix), "bin", "python"') -m venv ... (haven’t tested it but the premise is to try to get the non virtualenv python location)

[uranusjr]

I believe this is already a reported issue in virtualenv pypa/virtualenv#1095 (and venv, but the devs think virtualenv is responsible).

[techalchemy]

Closing for now in favor of the various upstream issues — thanks for the report!