python: Odd behaviour with sys.executable

[asottile]

Please note we will close your issue without comment if you delete, do not read or do not fill out the issue checklist below and provide ALL the requested information. If you repeatedly fail to use the issue template, we will block you from ever submitting issues to Homebrew again.

• are reporting a bug others will be able to reproduce and not asking a question. If you're not sure or want to ask a question do so on our Discourse: https://discourse.brew.sh
• have a problem with brew install (or upgrade, reinstall) a single, official formula (not cask)? If it's a general brew problem please file this issue at Homebrew/brew: https://github.com/Homebrew/brew/issues/new/choose. If it's a brew cask problem please file this issue at https://github.com/Homebrew/homebrew-cask/issues/new/choose. If it's a tap (e.g. Homebrew/homebrew-php) problem please file this issue at the tap.
• ran brew update and can still reproduce the problem?
• ran brew doctor, fixed all issues and can still reproduce the problem?
• ran brew gist-logs <formula> (where <formula> is the name of the formula that failed) and included the output link?
• if brew gist-logs didn't work: ran brew config and brew doctor and included their output with your issue?

To help us debug your issue please explain:

• What you were trying to do (and why)
• What happened (include command output)
• What you expected to happen
• Step-by-step reproduction instructions (by running brew install commands)

Here's a script which reproduces my problem:

#!/usr/bin/env bash
set -euxo pipefail

brew install python

rm -rf venv bin

BREW_PYTHON="/usr/local/bin/python3"
VENV="${PWD}/venv"
EXE="${VENV}/bin/python"

: make a virtualenv to have a separate python executable that acts normal
curl --silent https://asottile.github.io/get-virtualenv.py |
    "${BREW_PYTHON}" - "${VENV}" >& /dev/null

: write a small script which just prints what executable it ran from
mkdir bin
echo -e '#!'"${EXE}\nimport sys;print(sys.executable)" > bin/t
chmod +x bin/t

export PATH="${PWD}/bin:${PATH}"

: normal invocation of 't' works
test "$(t)" = "${EXE}"

: running t from a python subprocess is different though?
test "$(/usr/local/opt/python/libexec/bin/python -c 'import subprocess; subprocess.call(("t",))')" = "${EXE}"

Despite my script bin/t having a full shebang (here's what it ends up looking like):

#!/tmp/t/venv/bin/python
import sys;print(sys.executable)

When run as a subprocess from brew's python it is reporting an executable which is entirely different from the one being used.

I suspect this is due to this

Here's the output on my machine (script returns 1)

$ bash test.sh
+ brew install python
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/cask).
No changes to formulae.

Warning: python 3.7.0 is already installed and up-to-date
To reinstall 3.7.0, run `brew reinstall python`
+ rm -rf venv bin
+ BREW_PYTHON=/usr/local/bin/python3
+ VENV=/tmp/t/venv
+ EXE=/tmp/t/venv/bin/python
+ : make a virtualenv to have a separate python executable that acts normal
+ curl --silent https://asottile.github.io/get-virtualenv.py
+ /usr/local/bin/python3 - /tmp/t/venv
+ : write a small script which just prints what executable it ran from
+ mkdir bin
+ echo -e '#!/tmp/t/venv/bin/python\nimport sys;print(sys.executable)'
+ chmod +x bin/t
+ export 'PATH=/tmp/t/bin:/Users/asottile/.cargo/bin:/Users/asottile/bin:/usr/local/opt/findutils/libexec/gnubin:/usr/local/opt/coreutils/libexec/gnubin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public'
+ PATH='/tmp/t/bin:/Users/asottile/.cargo/bin:/Users/asottile/bin:/usr/local/opt/findutils/libexec/gnubin:/usr/local/opt/coreutils/libexec/gnubin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public'
+ : normal invocation of t works
++ t
+ test /tmp/t/venv/bin/python = /tmp/t/venv/bin/python
+ : running t from a python subprocess is different 'though?'
++ /usr/local/opt/python/libexec/bin/python -c 'import subprocess; subprocess.call(("t",))'
+ test /usr/local/Cellar/python/3.7.0/libexec/bin/python = /tmp/t/venv/bin/python

[asottile]

note this is a MCVE, this manifests in an actual issue for the pre-commit formula in the following way:

$ PATH=/usr/local/opt/python/libexec/bin/:$PATH .git/hooks/pre-commit
[INFO] Installing environment for https://github.com/asottile/add-trailing-comma.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: Command: ('/usr/local/Cellar/python/3.7.0/libexec/bin/python', '-mvirtualenv', '/Users/asottile/.cache/pre-commit/repoq6xyu2yv/py_env-python3.7', '-p', 'python3.7')
Return code: 1
Expected return code: 0
Output: (none)
Errors: 
    /usr/local/opt/python/bin/python3.7: No module named virtualenv
    

Check the log at /Users/asottile/.cache/pre-commit/pre-commit.log

pre-commit uses the value of sys.executable to accurately invoke virtualenv with the same python that is running pre-commit, however if this value is incorrect (pointing at an irrelevant python executable) this falls flat

[asottile]

I have a workaround which works for me 🤷‍♂️

[asottile]

I found this issue with more information: https://bugs.python.org/issue22490