Location warnings on 21.2 (contributor discussions)

[uranusjr]

This is an accompanying issue to #10151 on the distutils-sysconfig switch, so discussions on concrete solutions are not drowned by the reports.

I categorised the reports we gathered in the past day:

https://gist.github.com/uranusjr/c03c2c452ee8faa11db996c6b5a3999d

(Some reports uncover multiple issues so a few rows are duplicated under different categories.)

From what I can tell, there are four main problematic environments:

On macOS, pip install --user in a framework build

I think I made a mistake in #9844 and the skip_osx_framework_user_special_case check isn’t working properly. This should be easy to fix.

PR ready in #10215.

On MSYS, sysconfig’s lib path does not contain site-packages

This is… I don’t know, extremely weird. Sounds like a bug in msys? There’s only one ill-formatted report about this, and the reporter did not respond to my request to clarify. So I think I’ll just ignore this unless someone else comes up with a better report.

On Linux, sudo pip install -U pip a system-provided pip and then attempt to sudo pip install a package under the default prefix

Account for ~50% of the reports. Since Linux distros patch both Python and pip to return different paths for system and user-land tools, a user-installed upstream pip have trouble understanding the scheme. I already told Debian devs about this and IIRC they already updated the patches to accomodate the sysconfig switch, but in any case, since this can only be solved by distro devs, so I’m going to just try to detect the platform and suppress the warning. I think this is doable by checking whether distutils contains deb_system and unix_local on Debian and derivatives. Red Hat derivatives are more difficult to detect, I think the only way is to check whether platlib and purelib are in different locations (Red Hat patches platlib to use lib64 instead of lib).

PR ready as a part of #10217.

On Homebrew and Linuxbrew, pip install into the Brew-managed system prefix

Homebrew uses a distutils.cfg file to overwrite the install scheme (similar patch for Linuxbrew). Since distutils.cfg is going to be deprecated and eventually removed with distutils itself, Homebrew needs to re-think how they manage the install schemes. I already raised this issue to Homebrew. On pip’s side, since the entire distutils config file thing is going away, we should raise a deprecation warning when it is detected (and suppress the normal location mismatch warning). I’m thinking something like

DEPRECATION: Overwriting package installation paths with distutils configuration files is deprecated
and will be removed in Python 3.12. If you are seeing this under Homebrew or Linuxbrew, please 
see https://github.com/Homebrew/homebrew-core/issues/76621

Linuxbrew seems to have one additional setup that Homebrew does not have: #10151 (comment) But the root cause is likely the same (maybe just version differences in Brew?)

PR ready in #10218.

On Linux (potentially all POSIX), pip install into any site on a Python with pymalloc enabled

This is actually a bug in sysconfig. sysconfig’s purelib and platlib values fail to include sys.abiflag (the m thing in python3.6m if you’re not familiar with the topic; there are three suffixes in total IIRC). But Python removed those suffixes in 3.8 (I think) so there’s no point fixing sysconfig now (again, we are only going to switch to sysconfig when running on 3.10+). We should add some special cases to suppress the warning.

PR ready as a part of #10217.

[corneliusroemer]

The deprecation warning that brew users get is possibly non-ideal. I as an end-user have no idea what to do with it. I have no idea what the deprecation notice is about and the discussion that's referenced doesn't explain what to do. There, I'm being told to just wait for it to be fixed. So it's not clear what the point of the deprecation notice is. Homebrew maintainers say their policy is to ship broken software if it's provided broken.

Can you explain what you want the end user to do? Shall I switch away from using brew to install python/pip? Or hope that there'll be a fix. Or just not worry?

The notice can appear tens of times if you install a lot of dependencies, so it looks like a big issue. Maybe it's not, I don't know. But the current situation is, from a neutral user's perspective not good. Thanks!

Homebrew/homebrew-core#76621 (comment)

[mojotx]

The deprecation warning that brew users get is possibly non-ideal. I as an end-user have no idea what to do with it. I have no idea what the deprecation notice is about and the discussion that's referenced doesn't explain what to do. There, I'm being told to just wait for it to be fixed. So it's not clear what the point of the deprecation notice is. Homebrew maintainers say their policy is to ship broken software if it's provided broken.

Can you explain what you want the end user to do? Shall I switch away from using brew to install python/pip? Or hope that there'll be a fix. Or just not worry?

The notice can appear tens of times if you install a lot of dependencies, so it looks like a big issue. Maybe it's not, I don't know. But the current situation is, from a neutral user's perspective not good. Thanks!

Homebrew/homebrew-core#76621 (comment)

Also, there's no obvious way to silence that specific warning, except by using the -q/--quiet option twice, which also silences errors. In my opinion, if they're going to add this warning, there should be some way to silence it, at least in the short term.

[uranusjr]

I have responded to the Homebrew thread and hopefully we can move the conversation forward.

[ichard26]

In light of 698ccec and the completion of the sysconfig transition, do you think this issue is still worth keeping around @uranusjr ?

[uranusjr]

Let’s call this done.