A bunch of CI failures on main branch

[sobolevn]

The CI does not look good. There are multiple failures, they are unrelated, but all do happen on random commits.

Grouping them in one meta issue for now.

• asyncio failures:

======================================================================
FAIL: test_subprocess_shell (test.test_asyncio.test_events.KqueueEventLoopTests.test_subprocess_shell)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/cpython/cpython/Lib/test/test_asyncio/test_events.py", line 1813, in test_subprocess_shell
    self.assertEqual(proto.data[1].rstrip(b'\r\n'), b'Python')
AssertionError: b'' != b'Python'

CI link: https://github.com/python/cpython/actions/runs/3193117202/jobs/5211355431#step:7:3155
Commit: b44372e

• Altered environment on Windows:

 Traceback (most recent call last):
  File "D:\a\cpython\cpython\Lib\test\support\os_helper.py", line 483, in temp_dir
    yield path
  File "D:\a\cpython\cpython\Lib\test\support\os_helper.py", line 536, in temp_cwd
    yield cwd_dir
  File "D:\a\cpython\cpython\Lib\test\libregrtest\main.py", line 701, in main
    self._main(tests, kwargs)
  File "D:\a\cpython\cpython\Lib\test\libregrtest\main.py", line 758, in _main
    sys.exit(0)
SystemExit: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\a\cpython\cpython\Lib\test\support\__init__.py", line 203, in _force_run
    return func(*args)
           ^^^^^^^^^^^
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\\a\\cpython\\cpython\\build\\test_python_5740�\\test_python_worker_5604�'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\a\cpython\cpython\Lib\runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\a\cpython\cpython\Lib\runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "D:\a\cpython\cpython\Lib\test\__main__.py", line 2, in <module>
    main()
  File "D:\a\cpython\cpython\Lib\test\libregrtest\main.py", line 763, in main
    Regrtest().main(tests=tests, **kwargs)
  File "D:\a\cpython\cpython\Lib\test\libregrtest\main.py", line 695, in main
    with os_helper.temp_cwd(test_cwd, quiet=True):
  File "D:\a\cpython\cpython\Lib\contextlib.py", line 155, in __exit__
    self.gen.throw(value)
  File "D:\a\cpython\cpython\Lib\test\support\os_helper.py", line 534, in temp_cwd
    with temp_dir(path=name, quiet=quiet) as temp_path:
  File "D:\a\cpython\cpython\Lib\contextlib.py", line 155, in __exit__
    self.gen.throw(value)
  File "D:\a\cpython\cpython\Lib\test\support\os_helper.py", line 488, in temp_dir
    rmtree(path)
  File "D:\a\cpython\cpython\Lib\test\support\os_helper.py", line 445, in rmtree
    _rmtree(path)
  File "D:\a\cpython\cpython\Lib\test\support\os_helper.py", line 388, in _rmtree
    _waitfor(_rmtree_inner, path, waitall=True)
  File "D:\a\cpython\cpython\Lib\test\support\os_helper.py", line 333, in _waitfor
    func(pathname)
  File "D:\a\cpython\cpython\Lib\test\support\os_helper.py", line 385, in _rmtree_inner
    _force_run(fullname, os.rmdir, fullname)
  File "D:\a\cpython\cpython\Lib\test\support\__init__.py", line 214, in _force_run
    return func(*args)
           ^^^^^^^^^^^
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\\a\\cpython\\cpython\\build\\test_python_5740�\\test_python_worker_5604�'

== Tests result: FAILURE then SUCCESS ==

403 tests OK.

2 tests altered the execution environment:
    test_distutils test_peg_generator

1 re-run test:
    test_asyncio

CI link: https://github.com/python/cpython/actions/runs/3193117202/jobs/5211355655#step:6:2850
Commit: b44372e

• Env changed in buildbot

== Tests result: ENV CHANGED ==
410 tests OK.

1 test altered the execution environment:
    test_subprocess

Total duration: 8 min 25 sec
Tests result: ENV CHANGED
make: *** [buildbottest] Error 3

CI link: https://buildbot.python.org/all/#/builders/725/builds/2739
Commit: 23e83a8

[arhadthedev]

That's strange to have file access race emerged:

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\a\cpython\cpython\build\test_python_5740�\test_python_worker_5604�'

However, I'll create a reverting PR to stabilize the main and make the second attempt with all but asyncio tests manually disabled.

[sobolevn]

@arhadthedev the same error message is seen in #97641 🤔

[arhadthedev]

@sobolevn @vstinner I've prepared a full rollback of the two offending PRs:

• gh-97983: Revert "Lay the foundation for further work in asyncio.test_streams: port server cases to IsolatedAsyncioTestCase" #98015

Would you mind to merge it please? It's easier to revert everything back, debug the problem afterwards and unrevert the PR back.

[gvanrossum]

Merged, first thing in the morning. :-) Thanks for taking this seriously!

[vstinner]

2 tests altered the execution environment:
    test_distutils test_peg_generator

For these failures, see issue #93919: On Windows, test_distutils and test_peg_generator leak a temporary file: Microsoft.

[vstinner]

I created issue #98219 for "PermissionError: [WinError 32] The process cannot access the file because it is being used by another process".

[hauntsaninja]

Thanks for filing this! Looks like all three issues have been taken care of