pip freeze returns "Permission denied: 'hg'" after using pip install -e

[KevinGDialpad]

Environment

• pip version: 19.3.1 and above
• Python version: 3.7.7
• OS: macOS 10.14.6

Description
After using pip install -e to install a local package, pip freeze returns an error: PermissionError: [Errno 13] Permission denied: 'hg'.

I do not use Mercurial and do not have an hg executable on my machine.

It happens with pip 19.3.1 and above, when using -e.

It doesn't happen with pip 19.2.3, and it doesn't happen if I don't use -e. It doesn't happen with pip list.

Expected behavior
pip freeze should return the list of packages.

How to Reproduce
Create a simple package, install it using pip install -e, then run pip freeze.
It works with pip 19.2.3 but in 19.3.1 or later versions, pip freeze returns an error.

You can just copy/paste the following, which creates a virtual environment and does the pip install -e and the pip freeze on two different versions of pip:

cd
mkdir testdir
cd testdir/
virtualenv testenv
source testenv/bin/activate
mkdir my_plugin
echo "
from setuptools import setup

setup(
    name='my_plugin',
    py_modules=['my_plugin'],
    version='1.0.0',
)
" > my_plugin/setup.py
pip install pip==19.2.3
pip -V
pip install -e my_plugin/
pip freeze
pip uninstall -y my_plugin
pip freeze
pip install pip==19.3.1
pip -V
pip install -e my_plugin/
pip freeze
pip uninstall -y my_plugin
pip freeze

Output
Here's the output of the above commands, starting at the first pip -V:

$ pip -V
pip 19.2.3 from /Users/<username>/testdir/testenv/lib/python3.7/site-packages/pip (python 3.7)
$ pip install -e my_plugin/
Obtaining file:///Users/<username>/testdir/my_plugin
Installing collected packages: my-plugin
  Running setup.py develop for my-plugin
Successfully installed my-plugin
WARNING: You are using pip version 19.2.3, however version 20.2b1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ pip freeze
# Editable install with no version control (my-plugin==1.0.0)
-e /Users/<username>/testdir/my_plugin
$ pip uninstall -y my_plugin
Uninstalling my-plugin-1.0.0:
  Successfully uninstalled my-plugin-1.0.0
$ pip freeze
$ pip install pip==19.3.1
Collecting pip==19.3.1
  Using cached https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-19.3.1
$ pip -V
pip 19.3.1 from /Users/<username>/testdir/testenv/lib/python3.7/site-packages/pip (python 3.7)
$ pip install -e my_plugin/
Obtaining file:///Users/<username>/testdir/my_plugin
Installing collected packages: my-plugin
  Running setup.py develop for my-plugin
Successfully installed my-plugin
WARNING: You are using pip version 19.3.1; however, version 20.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ pip freeze
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/<username>/testdir/testenv/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 153, in _main
    status = self.run(options, args)
  File "/Users/<username>/testdir/testenv/lib/python3.7/site-packages/pip/_internal/commands/freeze.py", line 100, in run
    for line in freeze(**freeze_kwargs):
  File "/Users/<username>/testdir/testenv/lib/python3.7/site-packages/pip/_internal/operations/freeze.py", line 70, in freeze
    req = FrozenRequirement.from_dist(dist)
  File "/Users/<username>/testdir/testenv/lib/python3.7/site-packages/pip/_internal/operations/freeze.py", line 249, in from_dist
    req, editable, comments = get_requirement_info(dist)
  File "/Users/<username>/testdir/testenv/lib/python3.7/site-packages/pip/_internal/operations/freeze.py", line 188, in get_requirement_info
    vcs_backend = vcs.get_backend_for_dir(location)
  File "/Users/<username>/testdir/testenv/lib/python3.7/site-packages/pip/_internal/vcs/versioncontrol.py", line 234, in get_backend_for_dir
    if vcs_backend.controls_location(location):
  File "/Users/<username>/testdir/testenv/lib/python3.7/site-packages/pip/_internal/vcs/mercurial.py", line 149, in controls_location
    log_failed_cmd=False)
  File "/Users/<username>/testdir/testenv/lib/python3.7/site-packages/pip/_internal/vcs/versioncontrol.py", line 632, in run_command
    log_failed_cmd=log_failed_cmd)
  File "/Users/<username>/testdir/testenv/lib/python3.7/site-packages/pip/_internal/utils/subprocess.py", line 190, in call_subprocess
    stdout=subprocess.PIPE, cwd=cwd, env=env,
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
PermissionError: [Errno 13] Permission denied: 'hg'
$ pip uninstall -y my_plugin
Uninstalling my-plugin-1.0.0:
  Successfully uninstalled my-plugin-1.0.0
$ pip freeze
$ 

pip list still works. In fact, if I add pip list just before the problematic pip freeze in the chain of commands above, there is no error:

$ pip list
Package    Version
---------- -------
my-plugin  1.0.0  
pip        19.3.1 
setuptools 47.1.1 
wheel      0.34.2 
WARNING: You are using pip version 19.3.1; however, version 20.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ pip freeze
my-plugin==1.0.0

[uranusjr]

Is my_plugin managed by mercurial, and does your project root contain a .hg directory? I suspect this is a side effect of #7065; the Mercurial detection code path is much less trotted than Git due to popularity, and this may be hitting an edge case where Mercurial is expected but not actually being available.

What does it show if you run hg in the command line?

[KevinGDialpad]

So I don't have hg at all:

$ hg
-bash: hg: command not found
$ type hg
-bash: type: hg: not found

In my repro steps, there is no .hgfolder, either in ~/testdir/ or in ~.

I'm not sure what makes pip expect to find Mercurial.

Oh, I also found #7072. It's about Mercurial and pip freeze and was merged between 19.2.3 and 19.3.1, so it seems likely that the problem started there. I'll look more.

[uranusjr]

It’d be awesome if you could git bisect the exact commit that introduced the breakage. This is quite difficult to reproduce locally 😢

[KevinGDialpad]

I'd never heard of that tool! This is awesome.

I'm running it between a5b1a57 (19.2.3) and a5b1a57 (19.3.1), which I know as good and bad respectively.

This is the result:

c0809f4183cbf40e3fd69245095537456d74b97f is the first bad commit
commit c0809f4183cbf40e3fd69245095537456d74b97f
Author: Tony Beswick <[email protected]>
Date:   Tue Sep 24 11:46:02 2019 +1200

    Added a `controls_location` implementation to the `Mercurial` class, so it porperly detects that subdirectories are under mercurial control.

:040000 040000 6a58155df7ff4b3457d40c849c2c4f659df01db6 3d487b2c2fb13982d89e0291b91b964d9be8416a M	src

It points to commit c0809f4.

I'm not sure why this code would run since I don't have Mercurial. But it makes sense to me that if pip is trying to run that code, then the cls.run_command() fails. I think I'll need to dig into why pip thinks it needs to run code for Mercurial.

[uranusjr]

Another interesting thing is that the implementation does try to catch the error when hg is not installed (the BadCommand block) but your environment raises something it does not expect. I wonder why your system raises PermissionError instead of FileNotFoundError, which is the expected exception for unavailable commands.

[KevinGDialpad]

Yes that's really odd.
I'll do my best to track this down and understand what's wrong with my environment and see if pip can detect that. Thanks for your help so far!

[pradyunsg]

Is there anything actionable here? I'm sorry but I can't tell.

[MatthewRalston]

I'm experiencing the same problem with my project. @uranusjr were you able to resolve this?

I noticed that an egg was being made with a different name than my source code, due to an uneditted setup.py

Removing the egg allowed pip freeze to run properly.

[KevinGDialpad]

I haven't taken the time to figure out why my environment behaves in this way. I still have the problem but am still working around it by pinning pip to an earlier version.

I'm okay closing the ticket for now (if that's what you're asking), if I can reopen it when I have more information about the problem.

[chdsbd]

I encountered this issue recently. I worked around the issue by downgrading pip to 19.2.3.

[uranusjr]

It may make sense if we just treat PermissionError the same as FileNotFoundError in the version control code. The two exceptions both signal that the version control tool is not available, which is what we really care about, after all.

Another choice would be to perform our own command resolution logic (via shutil.which()) instead of relying the operating system to emit a FileNotFoundError for the subprocess. But I’m not quite sure if that’s feasible for pip right now (because Python 2).

[tderouin]

I had the same issue, pip-20.2.4

$ pip freeze
ERROR: Exception:
Traceback (most recent call last):
  File "/home/ec2-user/st-prod/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 228, in _main
    status = self.run(options, args)
  File "/home/ec2-user/st-prod/lib/python3.6/site-packages/pip/_internal/commands/freeze.py", line 101, in run
    for line in freeze(**freeze_kwargs):
  File "/home/ec2-user/st-prod/lib/python3.6/site-packages/pip/_internal/operations/freeze.py", line 67, in freeze
    req = FrozenRequirement.from_dist(dist)
  File "/home/ec2-user/st-prod/lib/python3.6/site-packages/pip/_internal/operations/freeze.py", line 252, in from_dist
    req, editable, comments = get_requirement_info(dist)
  File "/home/ec2-user/st-prod/lib/python3.6/site-packages/pip/_internal/operations/freeze.py", line 187, in get_requirement_info
    vcs_backend = vcs.get_backend_for_dir(location)
  File "/home/ec2-user/st-prod/lib/python3.6/site-packages/pip/_internal/vcs/versioncontrol.py", line 333, in get_backend_for_dir
    repo_path = vcs_backend.get_repository_root(location)
  File "/home/ec2-user/st-prod/lib/python3.6/site-packages/pip/_internal/vcs/mercurial.py", line 147, in get_repository_root
    log_failed_cmd=False,
  File "/home/ec2-user/st-prod/lib/python3.6/site-packages/pip/_internal/vcs/versioncontrol.py", line 774, in run_command
    log_failed_cmd=log_failed_cmd)
  File "/home/ec2-user/st-prod/lib/python3.6/site-packages/pip/_internal/vcs/versioncontrol.py", line 119, in call_subprocess
    cwd=cwd
  File "/usr/lib64/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/lib64/python3.6/subprocess.py", line 1364, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
PermissionError: [Errno 13] Permission denied: 'hg'

I had to install hg-git to fix it

sudo yum install hg-git -y

[uranusjr]

I’m even more confused to hear your fix worked. What on earth is going on.

[laurent-laporte-pro]

Hi, I am having the same issue with pip freeze on the following configuration:

Environment

• Pip version: 20.3.1 (2020-12-03)
• Python version: 2.7.6
• OS: macOS 10.14.6

Diagnostic

I have two user profiles for development (named "first" and "second" here). On the first profile, pip works without problem; on the second profile, I have the issue: when I run hg on the command line, I get:

second@imac-de-laurent-1 ~> hg
fish: Unknown command 'hg'

So, I took a look on the first profile, it appears that hg works!

first@imac-de-laurent-1 ~> hg
Mercurial Distributed SCM
[…]

The problem is that the hg command in a symbolic link to a user-level installation of SourceTree:

second@iMac-de-Laurent ~> ls -l /usr/local/bin/hg 
lrwxr-xr-x  1 first  admin  93 13 oct  2016 /usr/local/bin/hg -> /Users/first/Applications/SourceTree.app/Contents/Resources/mercurial_local/hg_local

So, if I launch this command from the second profile, I get a "permission denied" error:

second@iMac-de-Laurent ~> /usr/local/bin/hg
second@iMac-de-Laurent ~ [126]> 

=> exit code is 126

How to fix?

From the Pip's point of view, the hg command should be considered unavailable. So you should take into account that Permission denied [Errno 13] is a valid exception, and should catch it. But, it could be useful for the user (the developer) to emit a warning and alert him that he doesn't have access to this tool even if it is installed.

In my situation, I will try to uninstall SourceTree and reinstall it "globally".

[laurent-laporte-pro]

Once SourceTree uninstalled (and the broken symbolic link removed), I tried pip freeze again.
This time, I get:

second@imac-de-laurent-1 ~/w/my_project(262|M?)> pip freeze --verbose
Created temporary directory: /private/var/folders/lr/hjtkqzxd189dw_x7dk0493h80000gp/T/pip-ephem-wheel-cache-9hFWsh
Checking in /Users/second/workspace/my_project/src for .svn (svn)...
Checking in /Users/second/workspace/my_project/src for .git (git)...
Checking in /Users/second/workspace/my_project/src for .bzr (bzr)...
Checking in /Users/second/workspace/my_project/src for .hg (hg)...
could not determine if /Users/second/workspace/my_project/src is under hg control because hg is not available
No VCS found for editable requirement "my_project==3.1.26" in: '/Users/second/workspace/my_project/src'
...
[list of dependencies]

It's curious because my project is under Subversion, so the checking of .svn should have worked. Is it another problem?

[uranusjr]

rom the Pip's point of view, the hg command should be considered unavailable.

This is arguably a user issue because hg is available, but corrupted. pip can definitely catch PermissionError and emit a better error message, but probably should not pretend the command does not exist at all.

[uranusjr]

I’ll mark this as “awaiting PR” since it is pretty clear what we want to do. Feel free to try your hands on this and we can continue the discussion from there!

[laurent-laporte-pro]

OK @uranusjr , I'm working on it.

I will use an error message suggesting the developper to reinstall the 'hg' (or whatever) tool for all users (globally).

For instance: "Command hg is available, but you don't have permission to execute it - Can you ask an admin to reinstall hg for all users?"

[uranusjr]

I think we want to

#. Suggest installing for the current user. (If the system already has an hg available under another acount, maybe that’s what the user want?)
#. Suggest admin support to install for all users.
#. Suggest checking PATH environment variable (maybe the user does have a working hg, but masked by an invalid one).

This is probably be too much to fit in an error message, so we may need to create a paragraph or two in documentation (maybe a section under Installing from local packages), and point the error message to it. So something like

Command hg is found at [path], but you don't have permission to execute it. See possible solutions at [doc-url]

[laurent-laporte-pro]

Other issue

This is unlikely, but this error may be caused by a security rule (ACL) preventing commands from being executed on certain devices, such as a USB flash drive, the /tmp directory... On POSIX systems, you can use the setfacl command to modify the ACL rules of a file.

This problem is little known, and it is worth mentioning it.

Error message (BadCommand)

I suggest this error message:

No permission to execute {cls.name!r} - install it locally, globally (ask admin), or check your PATH. See possible solutions at https://pip.pypa.io/en/latest/reference/pip_freeze/#issues.

The documentation about the issue

This issue is only related to pip freeze command (other works well),
during requirements info extraction in freeze.get_requirement_info().

It is related to any VCS command, not only hg (VCS installation issue).

Here is what we can say about this issue in pip_freeze.rst (or user_guide.rst)

Fixing permission denied

No permission to execute 'cmd' [...] (where cmd is 'bzr', 'git', 'hg' or 'svn')

The VCS command exists, but you have no permission to execute it.

This error occurs, for instance, when the command is installed only for another user.
So, the current user don't have permission to execute the other user command.

To solve that issue, you can:

• install the command for yourself (local installation),
• ask admin support to install for all users (global installation),
• check and correct the PATH variable of your own environment,
• check the ACL (Access Control List) for this command.

Comment in code

Since developers are curious, it is also good to mention the solutions in the source code.