test_threading fails and crashes when rerun #108987
Labels
Comments
serhiy-storchaka
added
OS-windows
type-crash
|
Running just |
|
I'm able to reproduce the issue on Windows 11 with Python built in 32-bit mode. See It seems like the assertion error requires precise timing to be triggered. Output (reformatted for readability): |
|
The test_default_timeout() failured with the Example of my laptop with 12 thread (6 CPU cores): |
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
Sep 8, 2023
If a thread is created during Python finalization, the newly spawned thread will exit immediately. thread_run() calls PyEval_AcquireThread() which checks if the thread must exit. The problem is that tstate is dereferenced earlier in _PyThreadState_Bind() which lead to a crash most of the time.
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
Sep 8, 2023
Fix _thread.start_new_thread() race condition. If a thread is created during Python finalization, the newly spawned thread now exits immediately instead of trying to access freed memory and lead to a crash. thread_run() calls PyEval_AcquireThread() which checks if the thread must exit. The problem was that tstate was dereferenced earlier in _PyThreadState_Bind() which leads to a crash most of the time.
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
Sep 8, 2023
Fix _thread.start_new_thread() race condition. If a thread is created during Python finalization, the newly spawned thread now exits immediately instead of trying to access freed memory and lead to a crash. thread_run() calls PyEval_AcquireThread() which checks if the thread must exit. The problem was that tstate was dereferenced earlier in _PyThreadState_Bind() which leads to a crash most of the time.
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
Sep 8, 2023
Fix _thread.start_new_thread() race condition. If a thread is created during Python finalization, the newly spawned thread now exits immediately instead of trying to access freed memory and lead to a crash. thread_run() calls PyEval_AcquireThread() which checks if the thread must exit. The problem was that tstate was dereferenced earlier in _PyThreadState_Bind() which leads to a crash most of the time.
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
Sep 8, 2023
Fix _thread.start_new_thread() race condition. If a thread is created during Python finalization, the newly spawned thread now exits immediately instead of trying to access freed memory and lead to a crash. thread_run() calls PyEval_AcquireThread() which checks if the thread must exit. The problem was that tstate was dereferenced earlier in _PyThreadState_Bind() which leads to a crash most of the time. Move _PyThreadState_CheckConsistency() from thread_run() to _PyThreadState_Bind().
|
Oh, the GitHub Action Windows x64 job is also affected by this bug. Log reformatted to make them more readable (I removed NUL bytes, add empty lines): I had to download logs manually and remove null bytes, the two assertion errors are written in UTF-16 in a 8-bit console, and so the log is full of NUL bytes. I removed these NUL bytes. By the way, the regrtest final report is a little bit surprising. First, 2 tests failed:
Then, these 2 tests were run again. But test_threading crashed, regrtest failed to get the JSON from the output which is polluted by assertion errors. In that case, regrtest kills other worker processes, and stops immediately. The surprised part is that the final report lists It's not obvious that something was badly wrong in the re-run step. |
vstinner
added a commit
that referenced
this issue
Sep 11, 2023
Fix _thread.start_new_thread() race condition. If a thread is created during Python finalization, the newly spawned thread now exits immediately instead of trying to access freed memory and lead to a crash. thread_run() calls PyEval_AcquireThread() which checks if the thread must exit. The problem was that tstate was dereferenced earlier in _PyThreadState_Bind() which leads to a crash most of the time. Move _PyThreadState_CheckConsistency() from thread_run() to _PyThreadState_Bind().
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
Sep 11, 2023
…n#109135) Fix _thread.start_new_thread() race condition. If a thread is created during Python finalization, the newly spawned thread now exits immediately instead of trying to access freed memory and lead to a crash. thread_run() calls PyEval_AcquireThread() which checks if the thread must exit. The problem was that tstate was dereferenced earlier in _PyThreadState_Bind() which leads to a crash most of the time. Move _PyThreadState_CheckConsistency() from thread_run() to _PyThreadState_Bind(). (cherry picked from commit 517cd82)
vstinner
added a commit
that referenced
this issue
Sep 11, 2023
) (#109272) gh-108987: Fix _thread.start_new_thread() race condition (#109135) Fix _thread.start_new_thread() race condition. If a thread is created during Python finalization, the newly spawned thread now exits immediately instead of trying to access freed memory and lead to a crash. thread_run() calls PyEval_AcquireThread() which checks if the thread must exit. The problem was that tstate was dereferenced earlier in _PyThreadState_Bind() which leads to a crash most of the time. Move _PyThreadState_CheckConsistency() from thread_run() to _PyThreadState_Bind(). (cherry picked from commit 517cd82)
|
I modified regrtest to fail with exit code 5 when a test fails and then pass when re-run: new For example, a recent Windows10 job is now marked as WARNINGS (orange) because test_threading.test_default_timeout() failed and then passed when re-run in verbose mode in a fresh process. |
|
This issue is about 3 things:
I prefer to stick this issue to the crash. I created a new issue #109401 about the last points, random test failure and "Unraisable exception". The crash is now fixed in 3.11 and main branhches. I'm waiting until #109133 backport is merged into 3.12 to be able to backport #109135 to 3.12. @Yhg1s decided that this fix has to wait for Python 3.12.1, so once the 3.12 branch will be reopened after Python 3.12.0 final release ("3.12.0 final: Monday, 2023-10-02" says PEP 693). For now, I prefer to close this issue. |
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
Oct 4, 2023
…n#109135) Fix _thread.start_new_thread() race condition. If a thread is created during Python finalization, the newly spawned thread now exits immediately instead of trying to access freed memory and lead to a crash. thread_run() calls PyEval_AcquireThread() which checks if the thread must exit. The problem was that tstate was dereferenced earlier in _PyThreadState_Bind() which leads to a crash most of the time. Move _PyThreadState_CheckConsistency() from thread_run() to _PyThreadState_Bind(). (cherry picked from commit 517cd82)
vstinner
added a commit
that referenced
this issue
Oct 4, 2023
) (#110342) * gh-108987: Fix _thread.start_new_thread() race condition (#109135) Fix _thread.start_new_thread() race condition. If a thread is created during Python finalization, the newly spawned thread now exits immediately instead of trying to access freed memory and lead to a crash. thread_run() calls PyEval_AcquireThread() which checks if the thread must exit. The problem was that tstate was dereferenced earlier in _PyThreadState_Bind() which leads to a crash most of the time. Move _PyThreadState_CheckConsistency() from thread_run() to _PyThreadState_Bind(). (cherry picked from commit 517cd82) * gh-109795: `_thread.start_new_thread`: allocate thread bootstate using raw memory allocator (#109808) (cherry picked from commit 1b8f236) --------- Co-authored-by: Radislav Chugunov <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Crash report
First,
BarrierTests.test_default_timeoutintest_threadingfails, producing also several unraisable exception messages. They are produced even during running few following tests.https://github.com/python/cpython/actions/runs/6094370280/job/16535778004#step:5:103
After rerunning
test_threadingit crashes.https://github.com/python/cpython/actions/runs/6094370280/job/16535778004#step:5:1067
Note also UTF-16 in logs, but this is other issue.
It only happened in Windows (x86) run, the run on Windows (x64) passed successfully, as well as on other platforms. It may be 32-bit Windows specific or random.
Linked PRs
The text was updated successfully, but these errors were encountered: