[BUG] KeyError: 'unix_user' with setuptools 63.2.0

[mxmlnkn]

setuptools version

63.2.0

Python version

3.10

OS

Ubuntu 22.04

Additional environment information

No response

Description

I still run into this issue. I'm opening a new issue because I'm not sure whether the closed issue still receives any attention from the maintainers and because I'm not the only one able to still reproduce the problem.

I still have this problem on Ubuntu 22.04 with Python 3.10 and setuptools 63.2.0 installed.

I tried a lot of different versions of setuptools: 59,60,61,63 and none works. It works when using python3 setup.py develop --user instead of python3 -m pip install --user --editable.

Note that I also have to use the site.ENABLE_USER_SITE = "--user" in sys.argv[1:] workaround described in this pip issue. It seems to me like installing a development version becomes increasingly impossible.

Expected behavior

The error should not happen

How to Reproduce

python3 -m pip install --user --editable 'git+https://github.com/mxmlnkn/ratarmount.git@develop#egg=ratarmountcore[full]&egginfo=ratarmountcore&subdirectory=core'

Output

Obtaining ratarmountcore[full] from git+https://github.com/mxmlnkn/ratarmount.git@develop#egg=ratarmountcore[full]&egginfo=ratarmountcore&subdirectory=core
  Cloning https://github.com/mxmlnkn/ratarmount.git (to revision develop) to ./src/ratarmountcore
  Running command git clone --filter=blob:none --quiet https://github.com/mxmlnkn/ratarmount.git /media/d/Myself/projects/ratarmount/worktrees/1/core/src/ratarmountcore
  Running command git checkout -b develop --track origin/develop
  Switched to a new branch 'develop'
  Branch 'develop' set up to track remote branch 'develop' from 'origin' by rebasing.
  Resolved https://github.com/mxmlnkn/ratarmount.git to commit 78cea219605e351d4f286d2a9c4bd9bcdd5acd8d
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: indexed-bzip2<2.0,>=1.3.1 in /home/user/.local/lib/python3.10/site-packages (from ratarmountcore[full]) (1.3.2)
Requirement already satisfied: python-xz~=0.4.0 in /home/user/.local/lib/python3.10/site-packages (from ratarmountcore[full]) (0.4.0)
Requirement already satisfied: rarfile~=4.0 in /home/user/.local/lib/python3.10/site-packages (from ratarmountcore[full]) (4.0)
Requirement already satisfied: indexed-zstd<2.0,>=1.2.2 in /home/user/.local/lib/python3.10/site-packages (from ratarmountcore[full]) (1.5.0)
Requirement already satisfied: indexed-gzip<2.0,>=1.6.3 in /home/user/.local/lib/python3.10/site-packages (from ratarmountcore[full]) (1.6.13)
Installing collected packages: ratarmountcore
  Running setup.py develop for ratarmountcore
    error: subprocess-exited-with-error
    
    × python setup.py develop did not run successfully.
    │ exit code: 1
    ╰─> [29 lines of output]
        running develop
        /usr/lib/python3/dist-packages/setuptools/command/easy_install.py:158: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
          warnings.warn(
        Traceback (most recent call last):
          File "<string>", line 2, in <module>
          File "<pip-setuptools-caller>", line 34, in <module>
          File "/media/d/Myself/projects/ratarmount/worktrees/1/core/src/ratarmountcore/core/setup.py", line 14, in <module>
            setup()
          File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 153, in setup
            return distutils.core.setup(**attrs)
          File "/usr/lib/python3/dist-packages/setuptools/_distutils/core.py", line 148, in setup
            return run_commands(dist)
          File "/usr/lib/python3/dist-packages/setuptools/_distutils/core.py", line 163, in run_commands
            dist.run_commands()
          File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 967, in run_commands
            self.run_command(cmd)
          File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 985, in run_command
            cmd_obj.ensure_finalized()
          File "/usr/lib/python3/dist-packages/setuptools/_distutils/cmd.py", line 107, in ensure_finalized
            self.finalize_options()
          File "/usr/lib/python3/dist-packages/setuptools/command/develop.py", line 52, in finalize_options
            easy_install.finalize_options(self)
          File "/usr/lib/python3/dist-packages/setuptools/command/easy_install.py", line 270, in finalize_options
            self._fix_install_dir_for_user_site()
          File "/usr/lib/python3/dist-packages/setuptools/command/easy_install.py", line 400, in _fix_install_dir_for_user_site
            self.select_scheme(scheme_name)
          File "/usr/lib/python3/dist-packages/setuptools/command/easy_install.py", line 741, in select_scheme
            scheme = INSTALL_SCHEMES[name]
        KeyError: 'unix_user'
        [end of output]
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> [29 lines of output]
    running develop
    /usr/lib/python3/dist-packages/setuptools/command/easy_install.py:158: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    Traceback (most recent call last):
      File "<string>", line 2, in <module>
      File "<pip-setuptools-caller>", line 34, in <module>
      File "/media/d/Myself/projects/ratarmount/worktrees/1/core/src/ratarmountcore/core/setup.py", line 14, in <module>
        setup()
      File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 153, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib/python3/dist-packages/setuptools/_distutils/core.py", line 148, in setup
        return run_commands(dist)
      File "/usr/lib/python3/dist-packages/setuptools/_distutils/core.py", line 163, in run_commands
        dist.run_commands()
      File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 967, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 985, in run_command
        cmd_obj.ensure_finalized()
      File "/usr/lib/python3/dist-packages/setuptools/_distutils/cmd.py", line 107, in ensure_finalized
        self.finalize_options()
      File "/usr/lib/python3/dist-packages/setuptools/command/develop.py", line 52, in finalize_options
        easy_install.finalize_options(self)
      File "/usr/lib/python3/dist-packages/setuptools/command/easy_install.py", line 270, in finalize_options
        self._fix_install_dir_for_user_site()
      File "/usr/lib/python3/dist-packages/setuptools/command/easy_install.py", line 400, in _fix_install_dir_for_user_site
        self.select_scheme(scheme_name)
      File "/usr/lib/python3/dist-packages/setuptools/command/easy_install.py", line 741, in select_scheme
        scheme = INSTALL_SCHEMES[name]
    KeyError: 'unix_user'
    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

[mxmlnkn]

I skimmed over the last reply in the closed issue before opening this one but the proposed solution sudo apt remove python-setuptools does not work. It is very invasive and removes a lot of other packages and even pip, which makes installing setuptools harder. python3 -m ensurepip then warns me that I should use the system python instead:

ensurepip is disabled in Debian/Ubuntu for the system python.

Python modules for the system python are usually handled by dpkg and apt-get.

    apt install python3-<module name>

Install the python3-pip package to use pip itself.  Using pip together
with the system python might have unexpected results for any system installed
module, so use it on your own risk, or make sure to only use it in virtual
environments.

But, with this, I assume that it is more of a rollout issue of the fix.

And the question persists: why isn't the setuptools used that I have installed via pip?

> dpkg -l '*python3-setuptools*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                   Version      Architecture Description
+++-======================-============-============-=============================================
un  python3-setuptools     <none>       <none>       (no description available)
ii  python3-setuptools-whl 59.6.0-1.2   all          Python Distutils Enhancements (wheel package)

> python3 -m pip install --user setuptools
Requirement already satisfied: setuptools in /home/user/.local/lib/python3.10/site-packages (63.2.0)

[abravalheri]

Hi @mxmlnkn, thank you very much for reporting this issue.

Is there any chance you could test this against the latest setuptools release, v65.3.0?

Setuptools release cycle does not cover releasing patches for previous versions. If the bug is still present, any possible fixes will be added to the latest version.

Please also note that in v64+, setuptools implements PEP 660. This means that pip will be the tool copying the files to the final position on the disk. You can read more about that in https://setuptools.pypa.io/en/latest/userguide/development_mode.html. You can also opt-into the previous legacy behaviour by setting the environment variable: SETUPTOOLS_ENABLE_FEATURES="legacy-editable".

And the question persists: why isn't the setuptools used that I have installed via pip?

This might be related to https://github.com/pypa/pip/issues/6264.

[mxmlnkn]

Is there any chance you could test this against the latest setuptools release, v65.3.0?

Simply updating setuptools to 65.3.0 on my system does not help. However, setting up a venv and doing it inside there does help. So, I guess this issue is fixed in 65.3.0 and I will subscribe to pip #6254 instead.

Pip issue #6264 seems only marginally related though because I was not using any virtualenv. The mentioned hack of using --no-use-pep517 also does not work because my project specifies a build backend of setuptools.build_meta in pyproject.toml. My setup is:

sudo apt install python3-pip
python3-m pip install --user --upgrade setuptools
python3 -m pip install --editable 'git+https://github.com/mxmlnkn/ratarmount.git@develop#egg=ratarmountcore[full]&egginfo=ratarmountcore&subdirectory=core'

And then I was hoping that that would suffice to use the newer setuptools but it seems the system-installed version unexpectedly takes precedence.

I also tried increasing the requirement in pyproject.toml: requires = ["setuptools >= 65.3.0", "wheel"] but I guess it has no effect because it is not used for the outer build system call only for the inner one to build the specified package but I'm not welll-versed with the internals here.

Creating a normal venv and installing a newer setuptools inside there and installing the desired project there seems to work but it is cumbersome to use a venv to develop my project. It can have its advantages, though.

python3 -m venv testvenv
. ./testvenv/bin/activate
python3 -m pip install --upgrade setuptools
python3 -m pip install --editable 'git+https://github.com/mxmlnkn/ratarmount.git@develop#egg=ratarmountcore[full]&egginfo=ratarmountcore&subdirectory=core'

Working output:

Obtaining ratarmountcore[full] from git+https://github.com/mxmlnkn/ratarmount.git@develop#egg=ratarmountcore[full]&egginfo=ratarmountcore&subdirectory=core
  Cloning https://github.com/mxmlnkn/ratarmount.git (to revision develop) to ./testvenv/src/ratarmountcore
  Running command git clone --filter=blob:none --quiet https://github.com/mxmlnkn/ratarmount.git /media/d/Myself/projects/ratarmount/testvenv/src/ratarmountcore
  Running command git checkout -b develop --track origin/develop
  Switched to a new branch 'develop'
  Branch 'develop' set up to track remote branch 'develop' from 'origin' by rebasing.
  Resolved https://github.com/mxmlnkn/ratarmount.git to commit 78cea219605e351d4f286d2a9c4bd9bcdd5acd8d
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Collecting indexed-gzip<2.0,>=1.6.3
  Downloading indexed_gzip-1.6.13-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.8/3.8 MB 6.4 MB/s eta 0:00:00
Collecting rarfile~=4.0
  Downloading rarfile-4.0-py3-none-any.whl (28 kB)
Collecting python-xz~=0.4.0
  Downloading python_xz-0.4.0-py3-none-any.whl (20 kB)
Collecting indexed-bzip2<2.0,>=1.3.1
  Using cached indexed_bzip2-1.3.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.4 MB)
Collecting indexed-zstd<2.0,>=1.2.2
  Downloading indexed_zstd-1.5.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (591 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 591.1/591.1 KB 6.5 MB/s eta 0:00:00
Building wheels for collected packages: ratarmountcore
  Building editable for ratarmountcore (pyproject.toml) ... done
  Created wheel for ratarmountcore: filename=ratarmountcore-0.4.0-0.editable-py3-none-any.whl size=5433 sha256=fc41f502c8d642e93025a40572fdb868b1ae8ab9131562e54ff5367a7f1e59de
  Stored in directory: /tmp/pip-ephem-wheel-cache-qc3wirc5/wheels/7b/d4/7a/0c9e2a4aab863a5ecd7572b27dbeec506fee50ac57c1b70c12
Successfully built ratarmountcore
Installing collected packages: ratarmountcore, rarfile, indexed-zstd, indexed-gzip, indexed-bzip2, python-xz
Successfully installed indexed-bzip2-1.3.1 indexed-gzip-1.6.13 indexed-zstd-1.5.1 python-xz-0.4.0 rarfile-4.0 ratarmountcore-0.4.0

[mxmlnkn]

I found this comment and the proposed workaround of --no-build-isolation also works, i.e., in full:

python3 -m pip install --user --no-build-isolation --editable \
'git+https://github.com/mxmlnkn/ratarmount.git@develop#egg=ratarmountcore[full]&egginfo=ratarmountcore&subdirectory=core'

quite a tongue breaker but still easier to manage (with an alias) than a venv.

[abravalheri]

Pip issue #6264 seems only marginally related though because I was not using any virtualenv.

The issue is a bit long, but if you follow the discussion, you can see that there are other circumstances identified to trigger the same behaviour, not only in virtualenvs. One of these circumstances is Debian based systems, which I believe is the case you are describing.

I am afraid there is no action point from the setuptools side here. PEP 517 requires the build frontend (e.g. pip or PyPA's build tool) to use the version of setuptools listed in [build-system] requires (pyproject.toml). This is not something setuptools can do itself.

For this reason I think the best course of action is closing this issue and recommending anyone interested to follow https://github.com/pypa/pip/issues/6264.

[mxmlnkn]

Pip issue 6264 has been closed and upgrading to pip 22.3.1 seems to have fixed this problem :)