pytype conformance tests crash if no python3.11 executable is in PATH

[AlexWaygood]

I attempted to run the conformance tests locally using Python 3.12, following the instructions given in https://github.com/python/typing/blob/main/conformance/README.md#running-the-conformance-test-tool. The mypy, pyright and pyre tests all ran, but the pytype tests crashed with the following traceback:

Running tests for pytype
Traceback (most recent call last):
  File "/Users/alexw/dev/typing/conformance/.venv/lib/python3.12/site-packages/pytype/config.py", line 704, in _store_python_version
    self.output_options.python_exe = compiler.get_python_executable(version)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexw/dev/typing/conformance/.venv/lib/python3.12/site-packages/pytype/pyc/compiler.py", line 135, in get_python_executable
    raise PythonNotFoundError()
pytype.pyc.compiler.PythonNotFoundError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/alexw/dev/typing/conformance/src/main.py", line 258, in <module>
    main()
  File "/Users/alexw/dev/typing/conformance/src/main.py", line 251, in main
    run_tests(root_dir, type_checker, test_cases, skip_timing=options.skip_timing)
  File "/Users/alexw/dev/typing/conformance/src/main.py", line 30, in run_tests
    tests_output = type_checker.run_tests([file.name for file in test_cases])
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexw/dev/typing/conformance/src/type_checker.py", line 287, in run_tests
    options = pytype_config.Options.create(python_version=(3, 11), quick=True)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexw/dev/typing/conformance/.venv/lib/python3.12/site-packages/pytype/config.py", line 107, in create
    return cls(options)
           ^^^^^^^^^^^^
  File "/Users/alexw/dev/typing/conformance/.venv/lib/python3.12/site-packages/pytype/config.py", line 88, in __init__
    Postprocessor(names, opt_map, options, self).process()
  File "/Users/alexw/dev/typing/conformance/.venv/lib/python3.12/site-packages/pytype/config.py", line 587, in process
    node.processor(value)
  File "/Users/alexw/dev/typing/conformance/.venv/lib/python3.12/site-packages/pytype/config.py", line 706, in _store_python_version
    self.error("Need a valid python%d.%d executable in $PATH" % version)
  File "/Users/alexw/dev/typing/conformance/.venv/lib/python3.12/site-packages/pytype/config.py", line 594, in error
    raise PostprocessingError(message)
pytype.config.PostprocessingError: Need a valid python3.11 executable in $PATH

It sounded like pytype wanted Python 3.11, so I tried again, this time creating an environment with Python 3.11. But I didn't get nearly as far:

(311-env) ~/dev/typing/conformance/src (type-type)⚡ % python main.py
Traceback (most recent call last):
  File "/Users/alexw/dev/typing/conformance/src/main.py", line 258, in <module>
    main()
  File "/Users/alexw/dev/typing/conformance/src/main.py", line 230, in main
    assert sys.version_info >= (3, 12)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

[erictraut]

This runs fine for me on MacOS using Python 3.12. Which platform are you using?

Pytype is the hardest of the four type checkers to keep working in the conformance tests. It (historically) has frequently changed output data structures which required corresponding changes to the test code. It's also so far from being conformant that it's difficult to write and score conformance tests for it.

Google recently laid off the pytype team in the Bay Area. My understanding is that responsibility for its maintenance has been transferred to a team of engineers in Munich, but its future is not clear to me. Will it be enhanced over time, or is it in basic maintenance mode to support Google's internal development efforts? Is the intent for it to be conformant with the typing spec, or will it go in a different direction?

@h-joo, I noticed that you have recently submitted a few commits to the pytype project. Perhaps you could provide some clarity here? If not, is there some other contact at Google who could speak to the future of pytype?

[JelleZijlstra]

See also python/typeshed#11958 regarding the future maintenance of pytype.

As for Alex's original question, I think you need to have both an active Python 3.12 environment for the test suite harness, and a python3.11 executable in your PATH to install pytype onto. I think I ran into this error at some point too but managed to get it to work.

It would be nice if we could run the conformance tests in the CI for this repo so we're less dependent on local setups.

[AlexWaygood]

This runs fine for me on MacOS using Python 3.12. Which platform are you using?

I'm also using Python 3.12 on MacOS. Possibly more relevantly, both my Python 3.12 and Python 3.11 installations are managed using pyenv. I do not have a python3.11 executable on my PATH. I don't object to fiddling about with my Python installations a little bit in order to make it work, but we should probably document exactly what the requirements are to run the tests locally :-)

[h-joo]

@erictraut

Google recently laid off the pytype team in the Bay Area. My understanding is that responsibility for its maintenance has been transferred to a team of engineers in Munich, but its future is not clear to me. Will it be enhanced over time, or is it in basic maintenance mode to support Google's internal development efforts? Is the intent for it to be conformant with the typing spec, or will it go in a different direction?

@h-joo, I noticed that you have recently submitted a few commits to the pytype project. Perhaps you could provide some clarity here? If not, is there some other contact at Google who could speak to the future of pytype?

What you said is correct, I've newly joined the team and am ramping up in pytype. We plan to enhance this over time, as the majority of our python users are depending on pytype. We also plan to catch up with the new upcoming language features in 3.12, 3.13. As far as I understood, pytype's support is incomplete for 3.11 but it works, missing support for some new language features. For 3.12, the support isn't there as I understood it.

To be honest, I'm far away from having a good understanding of pytype at the moment, so although we have plans to improve it, it will take some time for the team to be able to catch up.

I tried running the conformance and I don't see the above error message. I used python3.12 on gLinux.

[AlexWaygood]

I fixed things up locally so that python3.11 resolves to a pyenv 3.11 shim that executes Python 3.11 in my shell; this fixed things for me. It should still be documented as a requirement that you need to have a python3.11 executable available locally in order to run the tests, so I'll leave this open for now.