Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Packages installed using filepaths on Windows mapped drives cannot be uninstalled and exhibit other buggy behavior #8598

Closed
paul-ko opened this issue Jul 17, 2020 · 6 comments
Labels
type: support User Support

Comments

@paul-ko
Copy link

paul-ko commented Jul 17, 2020

  • pip version: 20.1.1
  • Python version: 3.8.4
  • Operating system: Microsoft Windows Server 2012 R2 Standard

Summary

If you are working on a mapped drive on Windows, if you install a package from a file location (i.e., local repo), you will not be able to uninstall it. If the package was installed using -e, attempting to reinstall it will fail with an AssertionError as well. Based on traceback from that error, the problem appears to be rooted in a faulty comparison between the UNC representation of the path and a non-UNC representation of that same path.

Depending on how the install was done (editable vs. noneditable; relative path vs. absolute path) and what operation you're attempting (uninstall vs. install again), the behavior differs.

I saw these problem immediately after upgrading to the recent pip and Python versions listed above. I had not seen any of them on somewhat older versions, although I hadn't tried all of these combinations before - I did some additional testing to try to provide a fuller picture of what's going on. I do know that I've done at least some of these operations successfully on Python 3.7.3 and a fairly recent version of pip - almost certainly a 20.x.x version.

Details

Uninstall

When you attempt to uninstall the package, no matter how you installed it, you will get the output WARNING: Skipping <package name> because it is not installed. The output of pip freeze indicates that it is still installed. It doesn't appear possible to uninstall the package, which is really unfortunate if the user installed it to their user-level site-packages instead of to a virtual environment.

The behavior is the same whether you use a relative path or absolute path to specify the dependency.

"Reinstall"

When you run the exact same install command time, as if to upgrade/reinstall it, one of two things will happen:

  1. If the install is editable, the command will fail with an AssertionError which suggests the problem is that the UNC representation of the egg-link is being compared to the standard Windows path (i.e., starting with the mapped drive letter), and bombing out because it doesn't match.
  2. If the install is not editable, the command will succeed, and appears to actually install whatever version of the dependency is at the specified path.

The behavior is the same whether you use a relative path or absolute path to specify the dependency.

egg-link contents

Since the error traceback referenced the egglink, here are the contents of one of those that was created while investigating this problem:

K:\pk-a\tmp\pip-bug\dd-test-core
.

Demonstration

Preamble
# Show the mapped drive
PS K:\pk-a\tmp\pip-bug\envroot> get-psdrive | where name -eq K

Name           Used (GB)     Free (GB) Provider      Root                                               CurrentLocation
----           ---------     --------- --------      ----                                               ---------------
K                3137.40       3313.80 FileSystem    \\colossus\apps\data\src                           pk-a\tmp\pip-bug\envroot

# Show the parent directory.
PS K:\pk-a\tmp\pip-bug\envroot> ls ..


    Directory: K:\pk-a\tmp\pip-bug


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----        7/17/2020   1:23 PM                backup
d-----        7/17/2020  12:27 PM                dd-test-core
d-----        7/17/2020   1:21 PM                envroot
-a----        7/17/2020   1:15 PM           2482 notes.md
Editable + relative path
# Create the virtual env.
PS K:\pk-a\tmp\pip-bug\envroot> virtualenv editable-relative-env
created virtual environment CPython3.8.4.final.0-32 in 11014ms
  creator CPython3Windows(dest=\\colossus\apps\DATA\Src\pk-a\tmp\pip-bug\envroot\editable-relative-env, clear=False
, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\pk
-a\AppData\Local\pypa\virtualenv)
    added seed packages: pip==20.1.1, setuptools==49.2.0, wheel==0.34.2
  activators BashActivator,BatchActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

# Activate it.
PS K:\pk-a\tmp\pip-bug\envroot> . .\editable-relative-env\Scripts\activate

# Install the dependency.
(editable-relative-env) PS K:\pk-a\tmp\pip-bug\envroot> pip install -e ..\dd-test-core
Obtaining file:///K:/pk-a/tmp/pip-bug/dd-test-core
Collecting pyodbc>=4.0.27
  Using cached pyodbc-4.0.30-cp38-cp38-win32.whl (58 kB)
Processing c:\users\pk-a\appdata\local\pip\cache\wheels\b6\58\ba\905922546c0e429c3dfb6ce5da210dbadfe0670256febb5315
\pypika-0.37.16-py2.py3-none-any.whl
Collecting pyyaml>=5.1.2
  Using cached PyYAML-5.3.1-cp38-cp38-win32.whl (199 kB)
Installing collected packages: pyodbc, pypika, pyyaml, dd-test-core
  Running setup.py develop for dd-test-core
Successfully installed dd-test-core pyodbc-4.0.30 pypika-0.37.16 pyyaml-5.3.1

# Fail to reinstall it.
(editable-relative-env) PS K:\pk-a\tmp\pip-bug\envroot> pip install -e ..\dd-test-core
Obtaining file:///K:/pk-a/tmp/pip-bug/dd-test-core
Requirement already satisfied: pyodbc>=4.0.27 in \\colossus\apps\data\src\pk-a\tmp\pip-bug\envroot\editable-relativ
e-env\lib\site-packages (from dd-test-core==8.0.0) (4.0.30)
Requirement already satisfied: pypika>=0.35.15 in \\colossus\apps\data\src\pk-a\tmp\pip-bug\envroot\editable-relati
ve-env\lib\site-packages (from dd-test-core==8.0.0) (0.37.16)
Requirement already satisfied: pyyaml>=5.1.2 in \\colossus\apps\data\src\pk-a\tmp\pip-bug\envroot\editable-relative
-env\lib\site-packages (from dd-test-core==8.0.0) (5.3.1)
Installing collected packages: dd-test-core
  Attempting uninstall: dd-test-core
    Found existing installation: dd-test-core 8.0.0
ERROR: Exception:
Traceback (most recent call last):
  File "\\colossus\apps\DATA\Src\pk-a\tmp\pip-bug\envroot\editable-relative-env\lib\site-packages\pip\_internal\cli
\base_command.py", line 188, in _main
    status = self.run(options, args)
  File "\\colossus\apps\DATA\Src\pk-a\tmp\pip-bug\envroot\editable-relative-env\lib\site-packages\pip\_internal\cli
\req_command.py", line 185, in wrapper
    return func(self, options, args)
  File "\\colossus\apps\DATA\Src\pk-a\tmp\pip-bug\envroot\editable-relative-env\lib\site-packages\pip\_internal\com
mands\install.py", line 398, in run
    installed = install_given_reqs(
  File "\\colossus\apps\DATA\Src\pk-a\tmp\pip-bug\envroot\editable-relative-env\lib\site-packages\pip\_internal\req
\__init__.py", line 63, in install_given_reqs
    uninstalled_pathset = requirement.uninstall(
  File "\\colossus\apps\DATA\Src\pk-a\tmp\pip-bug\envroot\editable-relative-env\lib\site-packages\pip\_internal\req
\req_install.py", line 675, in uninstall
    uninstalled_pathset = UninstallPathSet.from_dist(dist)
  File "\\colossus\apps\DATA\Src\pk-a\tmp\pip-bug\envroot\editable-relative-env\lib\site-packages\pip\_internal\req
\req_uninstall.py", line 542, in from_dist
    assert (link_pointer == dist.location), (
AssertionError: Egg-link k:\pk-a\tmp\pip-bug\dd-test-core does not match installed location of dd-test-core (at \\c
olossus\apps\data\src\pk-a\tmp\pip-bug\dd-test-core)

# Fail to uninstall it.
(editable-relative-env) PS K:\pk-a\tmp\pip-bug\envroot> pip uninstall ddtestcore
WARNING: Skipping ddtestcore as it is not installed.

# Check what's installed.
(editable-relative-env) PS K:\pk-a\tmp\pip-bug\envroot> pip freeze
-e [email protected]:v3/org/EDW_git/dd-test-core@908be5e3e5473729f7601bf74827261ceaa1a7ed#eg
g=dd_test_core
pyodbc==4.0.30
PyPika==0.37.16
PyYAML==5.3.1
Editable absolute path
# Create the virtual env.
PS K:\pk-a\tmp\pip-bug\envroot> virtualenv editable-absolute-env
created virtual environment CPython3.8.4.final.0-32 in 10571ms
  creator CPython3Windows(dest=\\colossus\apps\DATA\Src\pk-a\tmp\pip-bug\envroot\editable-absolute-env, clear=Fals
, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\pk
-a\AppData\Local\pypa\virtualenv)
    added seed packages: pip==20.1.1, setuptools==49.2.0, wheel==0.34.2
  activators BashActivator,BatchActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

# Activate it.
PS K:\pk-a\tmp\pip-bug\envroot> . .\editable-absolute-env\Scripts\activate

# Install the dependency.
(editable-absolute-env) PS K:\pk-a\tmp\pip-bug\envroot> pip install -e K:\pk-a\tmp\pip-bug\dd-test-core
Obtaining file:///K:/pk-a/tmp/pip-bug/dd-test-core
Collecting pyodbc>=4.0.27
  Using cached pyodbc-4.0.30-cp38-cp38-win32.whl (58 kB)
Processing c:\users\pk-a\appdata\local\pip\cache\wheels\b6\58\ba\905922546c0e429c3dfb6ce5da210dbadfe0670256febb5315
\pypika-0.37.16-py2.py3-none-any.whl
Collecting pyyaml>=5.1.2
  Using cached PyYAML-5.3.1-cp38-cp38-win32.whl (199 kB)
Installing collected packages: pyodbc, pypika, pyyaml, dd-test-core
  Running setup.py develop for dd-test-core
Successfully installed dd-test-core pyodbc-4.0.30 pypika-0.37.16 pyyaml-5.3.1

# Fail to reinstall it.
(editable-absolute-env) PS K:\pk-a\tmp\pip-bug\envroot> pip install -e K:\pk-a\tmp\pip-bug\dd-test-core
Obtaining file:///K:/pk-a/tmp/pip-bug/dd-test-core
Requirement already satisfied: pyodbc>=4.0.27 in \\colossus\apps\data\src\pk-a\tmp\pip-bug\envroot\editable-absolut
e-env\lib\site-packages (from dd-test-core==8.0.0) (4.0.30)
Requirement already satisfied: pypika>=0.35.15 in \\colossus\apps\data\src\pk-a\tmp\pip-bug\envroot\editable-absolu
te-env\lib\site-packages (from dd-test-core==8.0.0) (0.37.16)
Requirement already satisfied: pyyaml>=5.1.2 in \\colossus\apps\data\src\pk-a\tmp\pip-bug\envroot\editable-absolute
-env\lib\site-packages (from dd-test-core==8.0.0) (5.3.1)
Installing collected packages: dd-test-core
  Attempting uninstall: dd-test-core
    Found existing installation: dd-test-core 8.0.0
ERROR: Exception:
Traceback (most recent call last):
  File "\\colossus\apps\DATA\Src\pk-a\tmp\pip-bug\envroot\editable-absolute-env\lib\site-packages\pip\_internal\cli
\base_command.py", line 188, in _main
    status = self.run(options, args)
  File "\\colossus\apps\DATA\Src\pk-a\tmp\pip-bug\envroot\editable-absolute-env\lib\site-packages\pip\_internal\cli
\req_command.py", line 185, in wrapper
    return func(self, options, args)
  File "\\colossus\apps\DATA\Src\pk-a\tmp\pip-bug\envroot\editable-absolute-env\lib\site-packages\pip\_internal\com
mands\install.py", line 398, in run
    installed = install_given_reqs(
  File "\\colossus\apps\DATA\Src\pk-a\tmp\pip-bug\envroot\editable-absolute-env\lib\site-packages\pip\_internal\req
\__init__.py", line 63, in install_given_reqs
    uninstalled_pathset = requirement.uninstall(
  File "\\colossus\apps\DATA\Src\pk-a\tmp\pip-bug\envroot\editable-absolute-env\lib\site-packages\pip\_internal\req
\req_install.py", line 675, in uninstall
    uninstalled_pathset = UninstallPathSet.from_dist(dist)
  File "\\colossus\apps\DATA\Src\pk-a\tmp\pip-bug\envroot\editable-absolute-env\lib\site-packages\pip\_internal\req
\req_uninstall.py", line 542, in from_dist
    assert (link_pointer == dist.location), (
AssertionError: Egg-link k:\pk-a\tmp\pip-bug\dd-test-core does not match installed location of dd-test-core (at \\c
olossus\apps\data\src\pk-a\tmp\pip-bug\dd-test-core)

# Fail to uninstall it.
(editable-absolute-env) PS K:\pk-a\tmp\pip-bug\envroot> pip uninstall ddtestcore
WARNING: Skipping ddtestcore as it is not installed.

# Check what's installed.
(editable-absolute-env) PS K:\pk-a\tmp\pip-bug\envroot> pip freeze
-e [email protected]:v3/org/EDW_git/dd-test-core@908be5e3e5473729f7601bf74827261ceaa1a7ed#eg
g=dd_test_core
pyodbc==4.0.30
PyPika==0.37.16
PyYAML==5.3.1
Absolute path, non-editable
# Create the virtual env.
PS K:\pk-a\tmp\pip-bug\envroot> virtualenv absolute-env
created virtual environment CPython3.8.4.final.0-32 in 11074ms
  creator CPython3Windows(dest=\\colossus\apps\DATA\Src\pk-a\tmp\pip-bug\envroot\absolute-env, clear=False, global=
False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\pk
-a\AppData\Local\pypa\virtualenv)
    added seed packages: pip==20.1.1, setuptools==49.2.0, wheel==0.34.2
  activators BashActivator,BatchActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

# Activate it.
PS K:\pk-a\tmp\pip-bug\envroot> . .\absolute-env\Scripts\activate

# Install the dependency.
(absolute-env) PS K:\pk-a\tmp\pip-bug\envroot> pip install K:\pk-a\tmp\pip-bug\dd-test-core
Processing k:\pk-a\tmp\pip-bug\dd-test-core
Collecting pyodbc>=4.0.27
  Using cached pyodbc-4.0.30-cp38-cp38-win32.whl (58 kB)
Processing c:\users\pk-a\appdata\local\pip\cache\wheels\b6\58\ba\905922546c0e429c3dfb6ce5da210dbadfe0670256febb5315
\pypika-0.37.16-py2.py3-none-any.whl
Collecting pyyaml>=5.1.2
  Using cached PyYAML-5.3.1-cp38-cp38-win32.whl (199 kB)
Building wheels for collected packages: dd-test-core
  Building wheel for dd-test-core (setup.py) ... done
  Created wheel for dd-test-core: filename=dd_test_core-8.0.0-py3-none-any.whl size=53075 sha256=ef0a83180b1c1eaf95447a9
415c42b2b7008bbb1aec55a419607c5f0ab746a9c
  Stored in directory: c:\users\pk-a\appdata\local\pip\cache\wheels\d5\06\98\4820418c0005b0a7a1feca201f54d973eb52b5
e4e83cecfb14
Successfully built dd-test-core
Installing collected packages: pyodbc, pypika, pyyaml, dd-test-core
Successfully installed dd-test-core-8.0.0 pyodbc-4.0.30 pypika-0.37.16 pyyaml-5.3.1

# Fail to uninstall it.
(absolute-env) PS K:\pk-a\tmp\pip-bug\envroot> pip uninstall ddtestcore
WARNING: Skipping ddtestcore as it is not installed.

# Check what's installed.
(absolute-env) PS K:\pk-a\tmp\pip-bug\envroot> pip freeze
dd-test-core @ file:///K:/pk-a/tmp/pip-bug/dd-test-core
pyodbc==4.0.30
PyPika==0.37.16
PyYAML==5.3.1
Relative, non-editable
# Create the virtual env.
PS K:\pk-a\tmp\pip-bug\envroot> virtualenv relative-env
created virtual environment CPython3.8.4.final.0-32 in 11099ms
  creator CPython3Windows(dest=\\colossus\apps\DATA\Src\pk-a\tmp\pip-bug\envroot\relative-env, clear=False, global=
False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\pk
-a\AppData\Local\pypa\virtualenv)
    added seed packages: pip==20.1.1, setuptools==49.2.0, wheel==0.34.2
  activators BashActivator,BatchActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

# Activate it.
PS K:\pk-a\tmp\pip-bug\envroot> . .\relative-env\Scripts\activate

# Install the dependency.
(relative-env) PS K:\pk-a\tmp\pip-bug\envroot> pip install ..\dd-test-core
Processing k:\pk-a\tmp\pip-bug\dd-test-core
Collecting pyodbc>=4.0.27
  Using cached pyodbc-4.0.30-cp38-cp38-win32.whl (58 kB)
Processing c:\users\pk-a\appdata\local\pip\cache\wheels\b6\58\ba\905922546c0e429c3dfb6ce5da210dbadfe0670256febb5315
\pypika-0.37.16-py2.py3-none-any.whl
Collecting pyyaml>=5.1.2
  Using cached PyYAML-5.3.1-cp38-cp38-win32.whl (199 kB)
Building wheels for collected packages: dd-test-core
  Building wheel for dd-test-core (setup.py) ... done
  Created wheel for dd-test-core: filename=dd_test_core-8.0.0-py3-none-any.whl size=53075 sha256=6d7e3b094687c2b25972b22
e50d0c0d1541b1121cfdfedda3e1c59bff77eb661
  Stored in directory: c:\users\pk-a\appdata\local\pip\cache\wheels\d5\06\98\4820418c0005b0a7a1feca201f54d973eb52b5
e4e83cecfb14
Successfully built dd-test-core
Installing collected packages: pyodbc, pypika, pyyaml, dd-test-core
Successfully installed dd-test-core-8.0.0 pyodbc-4.0.30 pypika-0.37.16 pyyaml-5.3.1

# Fail to uninstall it.
(relative-env) PS K:\pk-a\tmp\pip-bug\envroot> pip uninstall ddtestcore
WARNING: Skipping ddtestcore as it is not installed.

# Check what's installed.
(relative-env) PS K:\pk-a\tmp\pip-bug\envroot> pip freeze
dd-test-core @ file:///K:/pk-a/tmp/pip-bug/dd-test-core
pyodbc==4.0.30
PyPika==0.37.16
PyYAML==5.3.1
@triage-new-issues triage-new-issues bot added the S: needs triage Issues/PRs that need to be triaged label Jul 17, 2020
@paul-ko paul-ko changed the title Packages installed used relative paths on Windows mapped drives cannot be uninstalled and exhibit other buggy behavior Packages installed using filepaths on Windows mapped drives cannot be uninstalled and exhibit other buggy behavior Jul 17, 2020
@chrahunt
Copy link
Member

I suspect the non-editable cases are caused by the package name: we should execute pip uninstall dd_test_core (or pip uninstall dd-test-core, which would normalize to the same thing) instead of pip uninstall ddtestcore. Can you please give it a try?

For the editable cases, there may be something more interesting going on. Can you locate the dd_test_core.egg-link file in the virtualenv site-packages directory and provide its contents?

@chrahunt chrahunt added S: awaiting response Waiting for a response/more information type: support User Support labels Jul 18, 2020
@triage-new-issues triage-new-issues bot removed the S: needs triage Issues/PRs that need to be triaged label Jul 18, 2020
@paul-ko
Copy link
Author

paul-ko commented Jul 18, 2020

@chrahunt, you're absolutely right about the uninstall - that is pretty silly of me not to notice, and running the command correctly works. Thanks for the kind reminder.

Here are the egg-link contents:

PS K:\pk-a\tmp\pip-bug\envroot> cat .\editable-absolute-env\Lib\site-packages\dd-test-core.egg-link
K:\pk-a\tmp\pip-bug\dd-test-core
.

@no-response no-response bot removed the S: awaiting response Waiting for a response/more information label Jul 18, 2020
@uranusjr
Copy link
Member

Does uninstall work for editable as well? I would expect it to fail, since the reinstall failure happens during the uninstall step.

@paul-ko
Copy link
Author

paul-ko commented Jul 18, 2020

@uranusjr , you're right - I didn't think to check uninstalling a package that uses an editable path, and that does fail still:

PS K:\pk-a\tmp\pip-bug\envroot> . .\editable-absolute-env\Scripts\activate
(editable-absolute-env) PS K:\pk-a\tmp\pip-bug\envroot> pip freeze
-e [email protected]:v3/org/EDW_git/dd-test-core@c42d9bd0276c7676d982413c785f1da5c06c4048#eg
g=dd_test_core
pyodbc==4.0.30
PyPika==0.37.16
PyYAML==5.3.1
(editable-absolute-env) PS K:\pk-a\tmp\pip-bug\envroot> pip uninstall dd_test_core
Found existing installation: dd-test-core 8.0.0
ERROR: Exception:
Traceback (most recent call last):
  File "\\colossus\apps\DATA\Src\pk-a\tmp\pip-bug\envroot\editable-absolute-env\lib\site-packages\pip\_internal\cli
\base_command.py", line 188, in _main
    status = self.run(options, args)
  File "\\colossus\apps\DATA\Src\pk-a\tmp\pip-bug\envroot\editable-absolute-env\lib\site-packages\pip\_internal\com
mands\uninstall.py", line 85, in run
    uninstall_pathset = req.uninstall(
  File "\\colossus\apps\DATA\Src\pk-a\tmp\pip-bug\envroot\editable-absolute-env\lib\site-packages\pip\_internal\req
\req_install.py", line 675, in uninstall
    uninstalled_pathset = UninstallPathSet.from_dist(dist)
  File "\\colossus\apps\DATA\Src\pk-a\tmp\pip-bug\envroot\editable-absolute-env\lib\site-packages\pip\_internal\req
\req_uninstall.py", line 542, in from_dist
    assert (link_pointer == dist.location), (
AssertionError: Egg-link k:\pk-a\tmp\pip-bug\dd-test-core does not match installed location of dd-test-core (at \\c
olossus\apps\data\src\pk-a\tmp\pip-bug\dd-test-core)

@joAschauer
Copy link

Hi there,
I get the same AssertionError when trying to reinstall or uninstall a package which was installed in editable mode from a network drive within a conda environment on windows 10:

AssertionError: Egg-link h:\path_to_package\src does not match installed location of packagename (at \\server.address\homes\username\path_to_package\src)

Windows 10,
Python 3.9.7
pip version 22.0.3

Is there any workaround except from moving the code to a local drive and setting up a new environment? Will this issue be resolved if #9453 could be merged?

@pradyunsg
Copy link
Member

This should be resolved with #11011! :)

@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 8, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
type: support User Support
Projects
None yet
Development

No branches or pull requests

5 participants