zipapp virtual environment creation fails if zipapp path is symlinked #1949

andyneff · 2020-09-25T19:57:40Z

Issue

On a fresh new system, trying to use the zipapp to setup a virtualenv when the pyz files is referenced by a symlink, it fails saying KeyError: "There is no item named 'nv/create/via_global_ref/_virtualenv.py' in the archive"

But it works on my machine?

Yes, if you ever used the virtualenv (zipapp?) successfully, it creates files in ~/Library/Application Support/virtualenv (on macOS) or ~/.local/share/virtualenv/ (on Linux) and the existence of these files (unzip and wheel) makes the zipapp work. By deleting that directory, you should be able to reproduce the problem. This was first discovered on travis-ci, where the machine is new every time.

Environment

OS: macOS 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64.
Python 3.7.7. (from brew), also tried 3.7.9

Also reproduced on

OS: fedora:31
Python 3.7.9

Output of the virtual environment creation

bash-3.2$ (cd /var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/foo; python3 /var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz -vvv --with-traceback /var/folders/g0/944401t11gz1jl7vxt
nzqs_m0001yj/T/foo/pipenv-2020.6.2)
206 setup logging to NOTSET [DEBUG report:43]
219 find interpreter for spec PythonSpec(path=/usr/local/opt/python/bin/python3.7) [INFO builtin:44]
219 proposed PythonInfo(spec=CPython3.7.7.final.0-64, exe=/usr/local/opt/python/bin/python3.7, platform=darwin, version='3.7.7 (default, Mar 10 2020, 15:43:33) \n[Clang 11.0.0 (clang-1100.0.33.17)]', encoding_fs_io=utf-8-utf-8) [INFO builtin:50]
219 accepted PythonInfo(spec=CPython3.7.7.final.0-64, exe=/usr/local/opt/python/bin/python3.7, platform=darwin, version='3.7.7 (default, Mar 10 2020, 15:43:33) \n[Clang 11.0.0 (clang-1100.0.33.17)]', encoding_fs_io=utf-8-utf-8) [DEBUG builtin:52]
224 filesystem is not case-sensitive [DEBUG info:28]
263 create virtual environment via CPython3Posix(dest=/private/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/foo/pipenv-2020.6.2, clear=False, global=False) [INFO session:52]
263 write /private/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/foo/pipenv-2020.6.2/pyvenv.cfg [DEBUG pyenv_cfg:34]
263     home = /usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7 [DEBUG pyenv_cfg:38]
263     implementation = CPython [DEBUG pyenv_cfg:38]
263     version_info = 3.7.7.final.0 [DEBUG pyenv_cfg:38]
263     virtualenv = 20.0.31 [DEBUG pyenv_cfg:38]
263     include-system-site-packages = false [DEBUG pyenv_cfg:38]
264     base-prefix = /usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7 [DEBUG pyenv_cfg:38]
264     base-exec-prefix = /usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7 [DEBUG pyenv_cfg:38]
264     base-executable = /usr/local/opt/python/bin/python3.7 [DEBUG pyenv_cfg:38]
264 remove file /private/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/foo/pipenv-2020.6.2/bin/python [DEBUG _sync:38]
264 symlink /usr/local/opt/python/bin/python3.7 to /private/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/foo/pipenv-2020.6.2/bin/python [DEBUG _sync:44]
265 Attempting to acquire lock 4396469136 on /Users/a-andy/Library/Application Support/virtualenv/unzip/20.0.31/_virtualenv.py.lock [DEBUG filelock:270]
265 Lock 4396469136 acquired on /Users/a-andy/Library/Application Support/virtualenv/unzip/20.0.31/_virtualenv.py.lock [INFO filelock:274]
265 extract /var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/via_global_ref/_virtualenv.py to /Users/a-andy/Library/Application Support/virtualenv/unzip/20.0.31/_virtualenv.py [DEBUG zipapp:19]
267 Attempting to release lock 4396469136 on /Users/a-andy/Library/Application Support/virtualenv/unzip/20.0.31/_virtualenv.py.lock [DEBUG filelock:315]
267 Lock 4396469136 released on /Users/a-andy/Library/Application Support/virtualenv/unzip/20.0.31/_virtualenv.py.lock [INFO filelock:318]
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/__main__.py", line 168, in <module>
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/__main__.py", line 164, in run
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/__main__.py", line 16, in run
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/run/__init__.py", line 28, in cli_run
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/run/session.py", line 46, in run
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/run/session.py", line 53, in _create
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/creator.py", line 157, in run
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/via_global_ref/builtin/via_global_self_do.py", line 86, in create
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 88, in create
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 91, in install_patch
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/via_global_ref/builtin/cpython/cpython3.py", line 27, in env_patch_text
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 102, in env_patch_text
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/app_data/base.py", line 54, in ensure_extracted
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/app_data/via_disk_folder.py", line 81, in extract
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/util/zipapp.py", line 22, in extract

  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/zipfile.py", line 1431, in getinfo
    'There is no item named %r in the archive' % name)
KeyError: "There is no item named 'nv/create/via_global_ref/_virtualenv.py' in the archive"

Note: Notice /virtuale is missing from the KeyError, the exact length /private which is the difference between the TMPDIR and the real path of the TMPDIR on macOS

On Fedora, the last line becomes:

KeyError: "There is no item named '/virtualenv/create/via_global_ref/_virtualenv.py' in the archive"

Steps to reproduce

docker run -it --rm fedora:31
mkdir -p /tmp/foo/bar
ln -s /tmp/foo/bar /tmp/foo/bar2
curl -L -o /tmp/foo/bar/virtualenv.pyz https://bootstrap.pypa.io/virtualenv/3.7/virtualenv.pyz
python3 /tmp/foo/bar2/virtualenv.pyz /tmp/foo/blah

Here are some one-liners to reproduce the error

docker run -it --rm fedora:32 bash -c 'REAL=/tmp/foo/car; SYM=/tmp/foo/bbb2; mkdir -p "${REAL}"; ln -s "${REAL}" "${SYM}"; curl -L -o "${REAL}/virtualenv.pyz" https://bootstrap.pypa.io/virtualenv/3.8/virtualenv.pyz; python3 "${SYM}/virtualenv.pyz" /tmp/blah'

docker run -it --rm fedora:32 bash -c 'REAL=/tmp/foo/car; SYM=/tmp/foo/bbb2; mkdir -p "${REAL}"; ln -s "${REAL}" "${SYM}"; curl -L -o "${REAL}/virtualenv.pyz" https://bootstrap.pypa.io/virtualenv/3.8/virtualenv.pyz; python3 "${SYM}/virtualenv.pyz" /tmp/blah'

Not all values of REAL/SYM fail. It appears if the length of the symlink path and realpath differ, it fails. If they are the exact same length, no error occurs

Fail

REAL	SYM
/tmp/foo/bar	/tmp/bar/bar2
/tmp/foo/bar	/tmp/foo/bc
/tmp/foo/car	/tmp/foo/bb
/tmp/bb	/tmp/aaa

Does not fail

REAL	SYM
/tmp/foo	/tmp/bar
/tmp/foo/bar	/tmp/foo/abc
/tmp/foo/car	/tmp/foo/bbb

The text was updated successfully, but these errors were encountered:

Signed-off-by: Andy Neff <[email protected]>

Fix for pypa/virtualenv#1949

gaborbernat · 2020-10-08T05:32:24Z

Wouldn't this be fixed if you're using the copy method, instead of the symlink one? The symlink method cannot work if you're deleting the app-data folder from where we symlink. 🤔 ?

andyneff · 2020-10-08T14:50:35Z

@gaborbernat Are you saying that a user should have to add flags when calling the zipapp?

docker run -it --rm fedora:32 bash -c 'REAL=/tmp/foo/car; SYM=/tmp/foo/bbb2; mkdir -p "${REAL}"; ln -s "${REAL}" "${SYM}"; curl -L -o "${REAL}/virtualenv.pyz" https://bootstrap.pypa.io/virtualenv/3.8/virtualenv.pyz; python3 "${SYM}/virtualenv.pyz" --copies /tmp/blah'
docker run -it --rm fedora:32 bash -c 'REAL=/tmp/foo/car; SYM=/tmp/foo/bbb2; mkdir -p "${REAL}"; ln -s "${REAL}" "${SYM}"; curl -L -o "${REAL}/virtualenv.pyz" https://bootstrap.pypa.io/virtualenv/3.8/virtualenv.pyz; python3 "${SYM}/virtualenv.pyz" --symlink-app-data /tmp/blah'

both fail the same

gaborbernat · 2020-10-08T15:11:14Z

I wonder why the copy fails, it should not, but will look into it. If you have free time to do it, and you can do it that would be even better 👍

gaborbernat · 2023-06-27T03:14:38Z

Seems no longer an issue as no one reported it in 3 years.

petamas · 2023-07-19T15:34:10Z

@gaborbernat: please reopen, I have just ran into this issue with virtualenv 20.24.0 on MacOS Catalina. (Why I'm still using Catalina is a long story.)

It seems to me that it's important that the .pyz is referred to through the symlink, i.e. if I cd into the directory, and only pass the name of the .pyz, then it works, but if I use the full path (which contains a symlink), then it fails.

gaborbernat · 2023-07-19T15:35:50Z

Please fill in a PR to fix it, we do not plan to work on this ourselves.

petamas · 2023-07-19T15:37:58Z

Thanks for the quick reply! I have no idea where in the code the problem is, but I'll try to take a look.

(For now, I worked around it by calling os.path.realpath() on the path to the .pyz before calling it in my scripts.)

petamas · 2023-07-19T15:47:43Z

Upon investigation of the code, it seems like addibg os.path.realpath to this line would solve the issue:

virtualenv/src/virtualenv/util/zipapp.py

Line 26 in 328f965

full_path = os.path.abspath(str(full_path))

ROOT has this function called, while full_path does not, so ROOT is not necessarily a prefix of full_path, making sub_file contain something weird. This also explains @andyneff 's observation about path lengths.

I'll try fixing it and PRing the fix later this week or next week.

jamesnyao · 2024-01-17T00:08:04Z

I'm seeing this issue too when upgrading our vpython client from Python3.8 to Python3.11 (using virtualenv@fdfec65ff031997503fb409f365ee3aeb4c2c89f)

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\__main__.py", line 167, in <module>
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\__main__.py", line 163, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\__main__.py", line 16, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\__init__.py", line 30, in cli_run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\session.py", line 42, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\session.py", line 49, in _create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\creator.py", line 153, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\builtin\via_global_self_do.py", line 93, in create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 83, in create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 86, in install_patch
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 97, in env_patch_text
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\app_data\base.py", line 54, in ensure_extracted
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\app_data\via_disk_folder.py", line 78, in extract
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\util\zipapp.py", line 19, in extract
    import {module}
                   ^
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\zipfile.py", line 1473, in getinfo
    raise KeyError(
KeyError: "There is no item named 'irtualenv/create/via_global_ref/_virtualenv.py' in the archive"

bruceauyeung · 2024-02-26T12:34:11Z

I'm seeing this issue too when building electron
Python 3.11.8+chromium.31

$ '/home/testuser/.cache/.vpython-root/store/cpython+g6hihhltq68c5ujs5ph5u5rhnc/contents/bin/python3' '-B' '-I' '/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz' '--no-download' '--always-copy' '/home/testuser/.cache/.vpython-root/store/python_venv-478cu52ikjb86ilmh6apfpko0k/contents'
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/__main__.py", line 167, in <module>
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/__main__.py", line 163, in run
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/__main__.py", line 16, in run
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/run/__init__.py", line 30, in cli_run
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/run/session.py", line 42, in run
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/run/session.py", line 49, in _create
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/create/creator.py", line 153, in run
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/create/via_global_ref/builtin/via_global_self_do.py", line 93, in create
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 83, in create
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 86, in install_patch
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/create/via_global_ref/builtin/cpython/cpython3.py", line 25, in env_patch_text
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 97, in env_patch_text
  File "/home/testuser/.cache/.vpython-root/store/cpython+g6hihhltq68c5ujs5ph5u5rhnc/contents/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/app_data/base.py", line 54, in ensure_extracted
  File "/home/testuser/.cache/.vpython-root/store/cpython+g6hihhltq68c5ujs5ph5u5rhnc/contents/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/app_data/via_disk_folder.py", line 78, in extract
  File "/home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/virtualenv.pyz/virtualenv/util/zipapp.py", line 19, in extract
    import {module}
                   ^
  File "/home/testuser/.cache/.vpython-root/store/cpython+g6hihhltq68c5ujs5ph5u5rhnc/contents/lib/python3.11/zipfile.py", line 1484, in getinfo
    raise KeyError(
KeyError: "There is no item named 'lenv.pyz/virtualenv/create/via_global_ref/_virtualenv.py' in the archive"

bruceauyeung · 2024-02-27T01:09:49Z

I'm seeing this issue too when upgrading our vpython client from Python3.8 to Python3.11 (using virtualenv@fdfec65ff031997503fb409f365ee3aeb4c2c89f)

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\__main__.py", line 167, in <module>
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\__main__.py", line 163, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\__main__.py", line 16, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\__init__.py", line 30, in cli_run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\session.py", line 42, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\session.py", line 49, in _create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\creator.py", line 153, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\builtin\via_global_self_do.py", line 93, in create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 83, in create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 86, in install_patch
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 97, in env_patch_text
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\app_data\base.py", line 54, in ensure_extracted
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\app_data\via_disk_folder.py", line 78, in extract
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\util\zipapp.py", line 19, in extract
    import {module}
                   ^
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\zipfile.py", line 1473, in getinfo
    raise KeyError(
KeyError: "There is no item named 'irtualenv/create/via_global_ref/_virtualenv.py' in the archive"

did you fix/bypass your problem ?

bruceauyeung · 2024-02-29T03:43:18Z

I'm seeing this issue too when upgrading our vpython client from Python3.8 to Python3.11 (using virtualenv@fdfec65ff031997503fb409f365ee3aeb4c2c89f)

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\__main__.py", line 167, in <module>
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\__main__.py", line 163, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\__main__.py", line 16, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\__init__.py", line 30, in cli_run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\session.py", line 42, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\session.py", line 49, in _create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\creator.py", line 153, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\builtin\via_global_self_do.py", line 93, in create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 83, in create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 86, in install_patch
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 97, in env_patch_text
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\app_data\base.py", line 54, in ensure_extracted
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\app_data\via_disk_folder.py", line 78, in extract
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\util\zipapp.py", line 19, in extract
    import {module}
                   ^
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\zipfile.py", line 1473, in getinfo
    raise KeyError(
KeyError: "There is no item named 'irtualenv/create/via_global_ref/_virtualenv.py' in the archive"

did you fix/bypass your problem ?

#2613 merged after virtualenv-20.17.1 being released. So backporting those modifications to 20.17.1 can fix the problem. steps:

cd /home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/
mv virtualenv.pyz virtualenv.pyz.bak
unzip virtualenv.pyz.bak -d virtualenv
vi virtualenv/virtualenv/util/zipapp.py

replace contents in zipapp.py

full_path = os.path.abspath(str(full_path))
    sub_file = full_path[len(ROOT) + 1 :]

to

  full_path = os.path.realpath(os.path.abspath(str(full_path)))
  prefix = ROOT + os.sep
  assert full_path.startswith(prefix), f"full_path={full_path} should start with prefix={prefix}"
  sub_file = full_path[len(prefix) :]

python3 -m zipapp virtualenv

Zack4396 · 2024-03-01T06:00:57Z

Tried #1949 (comment) solution, and re-run gclient sync command, the errors are gone.

Thanks

josegal · 2024-04-09T16:07:36Z

I'm seeing this issue too when upgrading our vpython client from Python3.8 to Python3.11 (using virtualenv@fdfec65ff031997503fb409f365ee3aeb4c2c89f)

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\__main__.py", line 167, in <module>
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\__main__.py", line 163, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\__main__.py", line 16, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\__init__.py", line 30, in cli_run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\session.py", line 42, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\run\session.py", line 49, in _create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\creator.py", line 153, in run
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\builtin\via_global_self_do.py", line 93, in create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 83, in create
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 86, in install_patch
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\create\via_global_ref\api.py", line 97, in env_patch_text
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\app_data\base.py", line 54, in ensure_extracted
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\app_data\via_disk_folder.py", line 78, in extract
  File "C:\Users\CLOUDT~1\AppData\Local\Temp\vpython_bootstrap2852531825\packages\virtualenv-20.17.1\virtualenv.pyz\virtualenv\util\zipapp.py", line 19, in extract
    import {module}
                   ^
  File "D:\a\_work\1\infra\depot_tools\.cipd_bin\3.11\bin\Lib\zipfile.py", line 1473, in getinfo
    raise KeyError(
KeyError: "There is no item named 'irtualenv/create/via_global_ref/_virtualenv.py' in the archive"

did you fix/bypass your problem ?

#2613 merged after virtualenv-20.17.1 being released. So backporting those modifications to 20.17.1 can fix the problem. steps:

cd /home/testuser/.cache/.vpython-root/store/virtualenv+8orjkq78m3kenajlbpj4ev6eq4/contents/virtualenv-20.17.1/
mv virtualenv.pyz virtualenv.pyz.bak
unzip virtualenv.pyz.bak -d virtualenv
vi virtualenv/virtualenv/util/zipapp.py

replace contents in zipapp.py

full_path = os.path.abspath(str(full_path))
    sub_file = full_path[len(ROOT) + 1 :]

to

  full_path = os.path.realpath(os.path.abspath(str(full_path)))
  prefix = ROOT + os.sep
  assert full_path.startswith(prefix), f"full_path={full_path} should start with prefix={prefix}"
  sub_file = full_path[len(prefix) :]

python3 -m zipapp virtualenv

I just got this error on a fresh download of the chromium dev kit on ubuntu -- this workaround worked for me too but I had to move the virtualenv.pyz.bak file out of the way first... Thank you!!!

Use realpath for virtualenv.pyz to mitigate pypa/virtualenv#1949 Change-Id: I4ddfb78bc435f9791ccae5e87cf5c0da404a110a Reviewed-on: https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/5449147 Reviewed-by: Brian Ryner <[email protected]> Commit-Queue: Chenlin Fan <[email protected]> Auto-Submit: Chenlin Fan <[email protected]>

This reverts commit 988aaae. Reason for revert: Change is breaking luci-go to infra autoroller. https://ci.chromium.org/ui/p/infra/builders/try/infra-try-jammy-64/b8750527694180069633/overview Original change's description: > vpython: use realpath when invoking virtualenv > > Use realpath for virtualenv.pyz to mitigate pypa/virtualenv#1949 > > Change-Id: I4ddfb78bc435f9791ccae5e87cf5c0da404a110a > Reviewed-on: https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/5449147 > Reviewed-by: Brian Ryner <[email protected]> > Commit-Queue: Chenlin Fan <[email protected]> > Auto-Submit: Chenlin Fan <[email protected]> Change-Id: I37d27f0fd80d93f722ed000985fa5b95f51c8c6d No-Presubmit: true No-Tree-Checks: true No-Try: true Reviewed-on: https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/5458911 Commit-Queue: Rubber Stamper <[email protected]> Bot-Commit: Rubber Stamper <[email protected]> Auto-Submit: Randy Maldonado <[email protected]>

This is a reland of https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/5449147 with fix for python2.7. Use realpath for virtualenv.pyz to mitigate pypa/virtualenv#1949 Change-Id: I4bf2f527d1a90577d25194e05bb2de00bc1d71ef Reviewed-on: https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/5458472 Commit-Queue: Brian Ryner <[email protected]> Reviewed-by: Brian Ryner <[email protected]> Auto-Submit: Chenlin Fan <[email protected]>

andyneff added the bug label Sep 25, 2020

andyneff added a commit to VisionSystemsInc/docker_recipes that referenced this issue Sep 25, 2020

Fix for pypa/virtualenv#1949

95a3e77

Signed-off-by: Andy Neff <[email protected]>

andyneff added a commit to VisionSystemsInc/docker_recipes that referenced this issue Sep 28, 2020

Merge pull request #27 from VisionSystemsInc/venv_1949

7c3f883

Fix for pypa/virtualenv#1949

gaborbernat closed this as completed Oct 8, 2020

gaborbernat reopened this Oct 8, 2020

alealexpro100 mentioned this issue Apr 21, 2022

pycharm: fix creation of virtualenv ScoopInstaller/Extras#8359

Merged

1 task

gaborbernat added the help-wanted label Jun 27, 2023

gaborbernat closed this as completed Jun 27, 2023

gaborbernat reopened this Jul 19, 2023

gaborbernat assigned petamas Jul 19, 2023

petamas added a commit to petamas/virtualenv that referenced this issue Jul 20, 2023

Add assert to make issue pypa#1949 more apparent

fba4f0a

petamas added a commit to petamas/virtualenv that referenced this issue Jul 20, 2023

Fix pypa#1949

35c8e13

petamas added a commit to petamas/virtualenv that referenced this issue Jul 20, 2023

Add assert to make issue pypa#1949 more apparent

2fde5c4

petamas added a commit to petamas/virtualenv that referenced this issue Jul 20, 2023

Fix pypa#1949

4f615a9

petamas mentioned this issue Jul 20, 2023

Fix #1949: zipapp virtual environment creation fails if zipapp path is symlinked #2613

Closed

5 tasks

HandSonic mentioned this issue May 10, 2024

Fix #1949: zipapp virtual environment creation fails if zipapp path is symlinked #2720

Closed

5 tasks

HandSonic pushed a commit to HandSonic/virtualenv that referenced this issue May 11, 2024

Fix pypa#1949

610994c

HandSonic mentioned this issue May 11, 2024

Fix #1949: zipapp virtual environment creation fails if zipapp path is symlinked #2722

Merged

5 tasks

HandSonic added a commit to HandSonic/virtualenv that referenced this issue May 11, 2024

Merge branch 'main' into Fix-pypa#1949

8f79622

HandSonic added a commit to HandSonic/virtualenv that referenced this issue May 11, 2024

Merge remote-tracking branch 'origin/Fix-#1949' into Fix-pypa#1949

9fe0943

gaborbernat closed this as completed in 0a3816e May 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

zipapp virtual environment creation fails if zipapp path is symlinked #1949

zipapp virtual environment creation fails if zipapp path is symlinked #1949

andyneff commented Sep 25, 2020 •

edited

Loading

gaborbernat commented Oct 8, 2020

andyneff commented Oct 8, 2020

gaborbernat commented Oct 8, 2020

gaborbernat commented Jun 27, 2023

petamas commented Jul 19, 2023

gaborbernat commented Jul 19, 2023

petamas commented Jul 19, 2023

petamas commented Jul 19, 2023

jamesnyao commented Jan 17, 2024 •

edited

Loading

bruceauyeung commented Feb 26, 2024 •

edited

Loading

bruceauyeung commented Feb 27, 2024

bruceauyeung commented Feb 29, 2024 •

edited

Loading

Zack4396 commented Mar 1, 2024

josegal commented Apr 9, 2024

zipapp virtual environment creation fails if zipapp path is symlinked #1949

zipapp virtual environment creation fails if zipapp path is symlinked #1949

Comments

andyneff commented Sep 25, 2020 • edited Loading

Issue

But it works on my machine?

Environment

Output of the virtual environment creation

Steps to reproduce

gaborbernat commented Oct 8, 2020

andyneff commented Oct 8, 2020

gaborbernat commented Oct 8, 2020

gaborbernat commented Jun 27, 2023

petamas commented Jul 19, 2023

gaborbernat commented Jul 19, 2023

petamas commented Jul 19, 2023

petamas commented Jul 19, 2023

jamesnyao commented Jan 17, 2024 • edited Loading

bruceauyeung commented Feb 26, 2024 • edited Loading

bruceauyeung commented Feb 27, 2024

bruceauyeung commented Feb 29, 2024 • edited Loading

Zack4396 commented Mar 1, 2024

josegal commented Apr 9, 2024

andyneff commented Sep 25, 2020 •

edited

Loading

jamesnyao commented Jan 17, 2024 •

edited

Loading

bruceauyeung commented Feb 26, 2024 •

edited

Loading

bruceauyeung commented Feb 29, 2024 •

edited

Loading