test_concurrent_futures sometimes hangs indefinitely on Windows

[AlexWaygood]

Bug report

test_concurrent_futures appears to be hanging indefinitely on Windows in our CI. This has happened twice three times in two days, on three separate unrelated CI runs.

Two PRs:

• gh-92032: Add soft keywords to rlcompleter #92029 (comment)
• bpo-45897: Fix frozen-slotted dataclass bug #29895: https://github.com/python/cpython/runs/6270908903?check_suite_focus=true

test_concurrent_futures has also been running for 4hr30 (and counting) on Windows (x86) on this recent push to main:

• 9b027d4: https://github.com/python/cpython/runs/6269422000?check_suite_focus=true

Neither PR had anything to do with concurrent.futures (and the push to main didn't have anything to do with concurrent.futures either). For the first PR, test_concurrent_futures was hanging on Windows (x64) but passed on Windows (x86). For the second PR, the test has passed on Windows (x64) but is hanging on Windows (x86).

cc. @pitrou, @brianquinlan, as listed experts for concurrent.futures.

[AlexWaygood]

I've run python -m test test_concurrent_futures -v five times in succession on my Windows x64 machine, but haven't yet been able to reproduce this behaviour locally.

[AlexWaygood]

The most recent significant change to concurrent.futures was #31408, merged by @JelleZijlstra. The most recent significant change to test_concurrent_futures was #91600, by @gpshead. I've no idea if either change is related to this.

[gpshead]

fwiw my change to the test removed this logic that put a 0.1 second delay at the start of most every test (this is already one of the longest running tests in our suite), which was added over a decade ago without explanation and doesn't make logical sense:

    def _prime_executor(self):
        # Make sure that the executor is ready to do work before running the
        # tests. This should reduce the probability of timeouts in the tests.
        futures = [self.executor.submit(time.sleep, 0.1)
                   for _ in range(self.worker_count)]
        for f in futures:
            f.result()

The only thing code like that can do is hope to work around a logical flaw in tests or hide an actual race condition bug. so if there is a logic flaw or race condition somewhere... we'd be best off finding it as the real fix.

So far I've not observed timeout failures, but if anyone can reproduce them it'd be great to get a snapshot of what processes were stuck running and hanging on what so we can understand the root of the issue.

[pitrou]

So far I've not observed timeout failures, but if anyone can reproduce them it'd be great to get a snapshot of what processes were stuck running and hanging on what so we can understand the root of the issue.

If nothing bad happens on the buildbots then great :-)

[AlexWaygood]

I opened this issue because I saw the same CI failure three times in two days. I haven't seen it since, though, and haven't been able to reproduce it locally, so I'll close this for now. If it happens again, I'll reopen.

[neonene]

I saw this issue on 3.11 branch Windows (x64):
https://github.com/python/cpython/runs/6939506633?check_suite_focus=true

[neonene]

I am not able to reproduce locally as well. Would it be possible to inject something that shows what test case is still running or got canceled?