Rust doesn't build without "python" name being available

[upsuper]

Since Ubuntu 20.04, the name python no longer available by default without python-is-python2 package being installed. The build scripts should point to python2 instead.

[upsuper]

Or if the scripts have been python3-ready, it should probably instead point to python3?

[Mark-Simulacrum]

I guess mistakenly, when landing the python 3 preparation patch I assumed that python would be symlinked to the preferred python installation on most systems, but it seems like that's not proven out in practice. Do we have some sense whether there is some executable we can put in the x.py script shebang which would mostly just work on most systems? If python3 will do the trick, I'm down for approving a PR switching us over to that.

[mati865]

I'm fairly sure that calling python3 x.py should work.

[Mark-Simulacrum]

@mati865 Do you mean in general? Or specifically to fix it not working in the common case?

I agree that doing so fixes the bug but I'm not clear on whether that's the right approach. It's surprising to me that we're forced to recommend python3 over python2.7, basically.

[iago-lito]

@Mark-Simulacrum For what it's worth, python2.7 has been deprecated since January 1st, 2020. So I would argue that it's fine to recommend python3 over python2.7 now.

[mati865]

@Mark-Simulacrum in this specific issue AFAIK user can choose which Python to use by calling python2 x.py or python3 x.py instead of ./x.py even if /usr/bin/python doesn't exist.
The only general solution I can think of is creating x.sh that will probe for available Python versions.

[Mark-Simulacrum]

@iago-lito Yes, I'm aware, but I believe that many distros still ship and use python 2.

I think x.sh seems like overdoing it, and I do want the common case to work. With that in mind, I'll file a PR later today switching to python 3 as the default (though we will continue to support python 2.7).

[Lonami]

though we will continue to support python 2.7

Python 2 was supposed to be sunset on 2015 (as announced in 2008), but people still supporting it led to it being extended to 2020. I don't think it's a good idea to support it any further, is there any reason to do so?

[Mark-Simulacrum]

This is not the place to discuss it -- if you want further details than this comment, please ask on Zulip etc. But I personally do not believe that it is Rust compiler's build systems place to force people to upgrade unrelated software; the intent of the build system is to work as well as it can on as wide a range of base systems as possible, including those that only have python 2.

Filed #71841 for the fix to this issue.

[RalfJung]

Filed #71841 for the fix to this issue.

That PR got closed, if I understood correctly due to Windows compatibility issues.

Looks like there is no python binary name that actually works across platforms anymore? Windows doesn't have python3 and some Linux distros don't have python? What a mess.

[iago-lito]

Looks like there is no python binary name that actually works across platforms anymore? Windows doesn't have python3 and some Linux distros don't have python? What a mess.

@RalfJung maybe we should file this to python X)

[RalfJung]

I think #75034 is a duplicate of this.

[jyn514]

There is now a tool called x which will find the appropriate version of python and run it: https://rustc-dev-guide.rust-lang.org/building/how-to-build-and-run.html#running-xpy-slightly-more-conveniently. Would that meet this use case?

[ameliabradley]

I've updated README.md to explicitly detail that users need a python command available on their system, otherwise the x.py script will not run directly using the ./x.py format. I've also added details on how to work around the issue.

[ameliabradley]

There is now a tool called x which will find the appropriate version of python and run it: https://rustc-dev-guide.rust-lang.org/building/how-to-build-and-run.html#running-xpy-slightly-more-conveniently. Would that meet this use case?

Such a script would be ideal, except using x specifically as part of the build instructions adds rust as a requirement to run the bootstrapper, which might not be ideal.

[ChrisDenton]

Windows doesn't have python3

To be precise, it may have python3. Basically there are two installers, the classic one available from python.org and the modern one available through the Windows Store. The latter one has python3.exe whereas the former requires you to do py -3 to select python 3.

So to sum up the three standard options are:

• python isn't installed
• python3.exe is in the Path
• py.exe is in the Path

[jyn514]

@ChrisDenton does that mean that "python.exe is in the PATH" is uncommon? Would it be better for Windows users if we switched to /usr/bin/python3 in the shebang?

[ChrisDenton]

@jyn514 Ah, python.exe (without the 3) should be in the PATH, unless that option wasn't selected when using the python.org installer. It could be either version 2 or 3 depending which are installed and it what order.

[Walther]

(hoisting from the other issue, closed in favor of this one)

macOS Monterey 12.3 was released today.

The release notes include:

Python

Deprecations

• Python 2.7 was removed from macOS in this update. Developers should use Python 3 or an alternative language instead. (39795874)

The x.py tool used for the development process of the compiler starts with the line

#!/usr/bin/env python

On macOS Monterey 12.3, running env python results in

env: python: No such file or directory

It would probably be the time to make the script require Python 3, with

#!/usr/bin/env python3

---

This means that in addition to various linux distributions, macOS joins the club of python not being available as a name.

It was also mentioned that the main reason for not having changed to python3 was Windows not supporting that name. A related issue here.

While it is possible to work around this (e.g. by editing the local x.py script to mention python3, or creating a symlink, or other), this is an annoying papercut and it will be hitting more and more people. Currently at least all Ubuntu users, all macOS users on 12.3, more - need to add a workaround manually on their system.

Considering this issue has been open since 2020, what are the possible actions / next steps we could take to fix this properly for everyone?

[jyn514]

Considering this issue has been open since 2020, what are the possible actions / next steps we could take to fix this properly for everyone?

#94829

[Mark-Simulacrum]

I think the likely immediate next step if someone would like to make progress on this is to do a survey across at least tier-1 platforms (macOS, Windows, Linux, in no particular order), with likely some more variants of each (by install method, etc) in terms of the likely binaries available in a common PATH, and ease of getting other binaries/symlinks (e.g., python-is-python3 and similar packages).

A comment with that information would let us make an informed decision as to:

• Is there a clear majority around some name?
  • Or, is there a platform that we could push to change/add an executable that would bring us to a clear majority?

To be clear, this is not really about python 2 vs. python 3; we support both and I don't expect that to change in the immediate future. My recollection (but hard data would be good) is that just picking a python version is insufficient to guarantee a particular executable names availability, so I don't think making any choices there actually would help us. If it would, that's fine too.

(Edit: Yes, in the long run #94829 is likely at least a start to the future, though I'm not sure we'll ever fully drop x.py -- or some similar tooling.)

[Walther]

Hoisting from Zulip:

Edition Windows 11 Pro
Version 21H2
Installed on    ‎12/‎12/‎2021
OS build    22000.556
Experience  Windows Feature Experience Pack 1000.22000.556.0

On my Windows 11 machine, if i run python3 in either Powershell or CMD, the Windows Store automatically opens to the official Python 3.10 software page, published by Python Software Foundation.

After I click the Get button to install it, in both Powershell and CMD, running python3 -V results in Python 3.10.2.

After the installation, if I write a small file

#!/usr/bin/env python3
import time
print("Hello, world")
time.sleep(5)

and run .\hello.py in Powershell or hello.py in CMD, on first run I get the modal for "what do you want to use to open this file", if i select Python 3.10, it will launch a new window where the output is shown.

So at least on windows 11 with the python3 installed directly from windows store (as it automatically suggests if you try to run python3) the shebang of #!/usr/bin/env python3 seems to work perfectly fine!

I then uninstalled the version from the Windows Store, and installed using the newest installer from https://www.python.org/downloads/release/python-3102/ . I installed with the default options, without selecting the "add python 3.10 to PATH" option, as it was unselected by default.

Even with this installation, running the abovementioned .\hello.py works in Powershell and hello.py in CMD, with the shebang set to #!/usr/bin/env python3. Conversely, if I set the shebang to #!/usr/bin/env python, then it actually fails with an error of Python was not found - as in, the current shebang is broken even on Windows.

It has also been mentioned that some people use the py.exe wrapper. The documentation of shebang lines on windows says the following:

4.8.2. Shebang Lines

If the first line of a script file starts with #!, it is known as a “shebang” line. Linux and other Unix like operating systems have native support for such lines and they are commonly used on such systems to indicate how a script should be executed. This launcher allows the same facilities to be used with Python scripts on Windows and the examples above demonstrate their use.

To allow shebang lines in Python scripts to be portable between Unix and Windows, this launcher supports a number of ‘virtual’ commands to specify which interpreter to use. The supported virtual commands are:

/usr/bin/env python

/usr/bin/python

/usr/local/bin/python

python

For example, if the first line of your script starts with

#! /usr/bin/python

The default Python will be located and used. As many Python scripts written to work on Unix will already have this line, you should find these scripts can be used by the launcher without modification. If you are writing a new script on Windows which you hope will be useful on Unix, you should use one of the shebang lines starting with /usr.

Any of the above virtual commands can be suffixed with an explicit version (either just the major version, or the major and minor version). Furthermore the 32-bit version can be requested by adding “-32” after the minor version. I.e. /usr/bin/python2.7-32 will request usage of the 32-bit python 2.7.

Note especially: "Any of the above virtual commands can be suffixed with an explicit version (either just the major version, or the major and minor version)"

I tried using this wrapper too, and I get the same behavior: py hello.py in both Powershell and CMD with shebang of #!/usr/bin/env python3 works, #!/usr/bin/env python doesn't.

Based on this, it looks to me like the shebang of #!/usr/bin/env python3 should work even on Windows, both when installed via Windows Store as well as when installed via the official Python installer. It even looks like the current shebang of #!/usr/bin/env python is broken even on Windows, which has previously been held as the blocker for switching to #!/usr/bin/env python3.

---

I would be more than happy to create a tiny PR for this, if allowed!

[dtolnay]

I've proposed a possible solution in #98474. Please take a look.