Fix FileNotFoundError when updating cache on Windows

[danyeaw]

• I have added an entry to docs/changelog.md

No changelog entry since fix isn't user facing (only impacts running the test suite).

Summary of changes

This PR fixes a FileNotFoundError on Windows caused by trying to call python3 directly in the update_package_cache module. I also made a couple of minor refactorings in adjacent areas of the code to help keep things nicer than I found it.

Test plan

On Windows, all tests are failing without this change. I noticed that the GitHub Action code to use a persistent package cache was created on the same day as this bug was introduced. Is it possible that the cache on GitHub Actions is 2 years old?

Would it be worth adding a hash of the pyproject.toml file to force at least an occasional refresh of the cache?

command(s) to exercise these changes

rm -r .pipx_test/package_cache
nox -s tests

Full traceback for a test:

PS C:\Users\dan\Projects\pipx> nox -x -s tests
nox > Running session tests-3.7
nox > Missing interpreters will error by default on CI systems.
nox > Session tests-3.7 skipped: Python interpreter 3.7 not found.
nox > Running session tests-3.8
nox > Re-using existing virtual environment at .nox\tests-3-8.
nox > python -m pip install --upgrade pip
Requirement already satisfied: pip in c:\users\dan\projects\pipx\.nox\tests-3-8\lib\site-packages (23.1.2)
nox > python -m pip install wheel
nox > pip wheel '--wheel-dir=C:\Users\dan\Projects\pipx\.nox\tests-3-8\prebuild_wheels' jupyter==1.0.0
nox > python -m pip install -e . pytest pytest-cov
nox > python -m pip install pypiserver
Updating local tests package spec file cache...
nox > python scripts/update_package_cache.py 'testdata\tests_packages' '.pipx_tests\package_cache'
Using the following file to specify needed package files:
    testdata\tests_packages\win-python3.8.txt
Ensuring the following directory contains necessary package files:
    .pipx_tests\package_cache\3.8
MISSING FILES: 0
EXISTING (found) FILES: 179
LEFTOVER (unused) FILES: 0
nox > pytest --cov=pipx --cov-report= tests
==================================================================== test session starts ====================================================================
platform win32 -- Python 3.8.10, pytest-7.3.2, pluggy-1.0.0
rootdir: C:\Users\dan\Projects\pipx
configfile: pyproject.toml
plugins: cov-4.1.0
collected 360 items / 124 deselected / 236 selected                                                                                                          

tests\test_animate.py EEEEEEEEE

========================================================================== ERRORS =========================================================================== 
______________________________________________________ ERROR at setup of test_delay_suppresses_output _______________________________________________________ 

request = <SubRequest 'pipx_local_pypiserver' for <Function test_delay_suppresses_output>>

    @pytest.fixture(scope="session", autouse=True)
    def pipx_local_pypiserver(request):
        """Starts local pypiserver once per session unless --net-pypiserver was
        passed to pytest"""
        if request.config.option.net_pypiserver:
            # need both yield and return because other codepath has both
            yield
            return

        pipx_cache_dir = (
            request.config.invocation_params.dir / PIPX_TESTS_DIR / "package_cache"
        )
        check_test_packages_cmd = [
            "python3",
            "scripts/update_package_cache.py",
            "--check-only",
            str(PIPX_TESTS_PACKAGE_LIST_DIR),
            str(pipx_cache_dir),
        ]
        update_test_packages_cmd = [
            "python3",
            "scripts/update_package_cache.py",
            str(PIPX_TESTS_PACKAGE_LIST_DIR),
            str(pipx_cache_dir),
        ]
>       check_test_packages_process = subprocess.run(check_test_packages_cmd)

tests\conftest.py:102:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
..\..\AppData\Local\Programs\Python\Python38\lib\subprocess.py:493: in run
    with Popen(*popenargs, **kwargs) as process:
..\..\AppData\Local\Programs\Python\Python38\lib\subprocess.py:858: in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <subprocess.Popen object at 0x000002B189909430>
args = 'python3 scripts/update_package_cache.py --check-only testdata\\tests_packages C:\\Users\\dan\\Projects\\pipx\\.pipx_tests\\package_cache'
executable = None, preexec_fn = None, close_fds = True, pass_fds = (), cwd = None, env = None
startupinfo = <subprocess.STARTUPINFO object at 0x000002B189909400>, creationflags = 0, shell = False, p2cread = -1, p2cwrite = -1, c2pread = -1
c2pwrite = -1, errread = -1, errwrite = -1, unused_restore_signals = True, unused_start_new_session = False

    def _execute_child(self, args, executable, preexec_fn, close_fds,
                       pass_fds, cwd, env,
                       startupinfo, creationflags, shell,
                       p2cread, p2cwrite,
                       c2pread, c2pwrite,
                       errread, errwrite,
                       unused_restore_signals, unused_start_new_session):
        """Execute program (MS Windows version)"""

        assert not pass_fds, "pass_fds not supported on Windows."

        if isinstance(args, str):
            pass
        elif isinstance(args, bytes):
            if shell:
                raise TypeError('bytes args is not allowed on Windows')
            args = list2cmdline([args])
        elif isinstance(args, os.PathLike):
            if shell:
                raise TypeError('path-like args is not allowed when '
                                'shell is true')
            args = list2cmdline([args])
        else:
            args = list2cmdline(args)

        if executable is not None:
            executable = os.fsdecode(executable)

        # Process startup details
        if startupinfo is None:
            startupinfo = STARTUPINFO()
        else:
            # bpo-34044: Copy STARTUPINFO since it is modified above,
            # so the caller can reuse it multiple times.
            startupinfo = startupinfo.copy()

        use_std_handles = -1 not in (p2cread, c2pwrite, errwrite)
        if use_std_handles:
            startupinfo.dwFlags |= _winapi.STARTF_USESTDHANDLES
            startupinfo.hStdInput = p2cread
            startupinfo.hStdOutput = c2pwrite
            startupinfo.hStdError = errwrite

        attribute_list = startupinfo.lpAttributeList
        have_handle_list = bool(attribute_list and
                                "handle_list" in attribute_list and
                                attribute_list["handle_list"])

        # If we were given an handle_list or need to create one
        if have_handle_list or (use_std_handles and close_fds):
            if attribute_list is None:
                attribute_list = startupinfo.lpAttributeList = {}
            handle_list = attribute_list["handle_list"] = \
                list(attribute_list.get("handle_list", []))

            if use_std_handles:
                handle_list += [int(p2cread), int(c2pwrite), int(errwrite)]

            handle_list[:] = self._filter_handle_list(handle_list)

            if handle_list:
                if not close_fds:
                    warnings.warn("startupinfo.lpAttributeList['handle_list'] "
                                  "overriding close_fds", RuntimeWarning)

                # When using the handle_list we always request to inherit
                # handles but the only handles that will be inherited are
                # the ones in the handle_list
                close_fds = False

        if shell:
            startupinfo.dwFlags |= _winapi.STARTF_USESHOWWINDOW
            startupinfo.wShowWindow = _winapi.SW_HIDE
            comspec = os.environ.get("COMSPEC", "cmd.exe")
            args = '{} /c "{}"'.format (comspec, args)

        if cwd is not None:
            cwd = os.fsdecode(cwd)

        sys.audit("subprocess.Popen", executable, args, cwd, env)

        # Start the process
        try:
>           hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
                                     # no special security
                                     None, None,
                                     int(not close_fds),
                                     creationflags,
                                     env,
                                     cwd,
                                     startupinfo)
E                                    FileNotFoundError: [WinError 2] The system cannot find the file specified