Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-107219: Fix concurrent.futures terminate_broken() #108974

Merged
merged 1 commit into from
Sep 6, 2023

Conversation

vstinner
Copy link
Member

@vstinner vstinner commented Sep 6, 2023

Fix a race condition in _ExecutorManagerThread.terminate_broken(): ignore the InvalidStateError on future.set_exception(). It can happen if the future is cancelled before the caller.

Moreover, test_crash_big_data() now waits explicitly until the executor completes.

Fix a race condition in _ExecutorManagerThread.terminate_broken():
ignore the InvalidStateError on future.set_exception(). It can happen
if the future is cancelled before the caller.

Moreover, test_crash_big_data() now waits explicitly until the
executor completes.
@vstinner
Copy link
Member Author

vstinner commented Sep 6, 2023

This change doesn't fix the most annoying bug of issue #107219. test_crash_big_data() still hangs on Windows. But it fix this error:

test_crash_big_data (test.test_concurrent_futures.ProcessPoolSpawnExecutorDeadlockTest.test_crash_big_data) ... Warning -- Uncaugh
t thread exception: InvalidStateError
Exception in thread Thread-9:
Traceback (most recent call last):
  File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\threading.py", line 1059, in _bootstrap_inner
    self.run()
  File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\concurrent\futures\process.py", line 344, in run
    self.terminate_broken(cause)
  File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\concurrent\futures\process.py", line 492, in terminate_broken
    work_item.future.set_exception(bpe)
  File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\concurrent\futures\_base.py", line 559, in set_exception
    raise InvalidStateError('{}: {!r}'.format(self._state, self))
concurrent.futures._base.InvalidStateError: CANCELLED: <Future at 0x2cf561cf260 state=cancelled>
0.55s Warning -- threading_cleanup() failed to cleanup 1 threads (count: 1, dangling: 2)
Warning -- Dangling thread: <_MainThread(MainThread, started 11880)>
Warning -- Dangling thread: <Thread(QueueFeederThread, started daemon 4112)>
ok

link: #107219 (comment)

@vstinner
Copy link
Member Author

vstinner commented Sep 6, 2023

cc @pitrou

@vstinner
Copy link
Member Author

vstinner commented Sep 6, 2023

This change doesn't fix the most annoying bug of issue #107219. test_crash_big_data() still hangs on Windows.

If it wasn't obvious enough, the test decided to hang on Windows x64 CI :-)

0:41:15 load avg: 0.02 [462/462/3] test.test_concurrent_futures.test_deadlock process crashed (Exit code 1)
Timeout (0:20:00)!
Thread 0x00000c90 (most recent call first):
  File "D:\a\cpython\cpython\Lib\multiprocessing\connection.py", line 282 in _send_bytes
  File "D:\a\cpython\cpython\Lib\multiprocessing\connection.py", line 199 in send_bytes
  File "D:\a\cpython\cpython\Lib\multiprocessing\queues.py", line 246 in _feed
  File "D:\a\cpython\cpython\Lib\threading.py", line 996 in run
  File "D:\a\cpython\cpython\Lib\threading.py", line 1059 in _bootstrap_inner
  File "D:\a\cpython\cpython\Lib\threading.py", line 1016 in _bootstrap

Thread 0x00001b78 (most recent call first):
  File "D:\a\cpython\cpython\Lib\threading.py", line 1153 in _wait_for_tstate_lock
  File "D:\a\cpython\cpython\Lib\threading.py", line 1133 in join
  File "D:\a\cpython\cpython\Lib\multiprocessing\queues.py", line 199 in _finalize_join
  File "D:\a\cpython\cpython\Lib\multiprocessing\util.py", line 224 in __call__
  File "D:\a\cpython\cpython\Lib\multiprocessing\queues.py", line 151 in join_thread
  File "D:\a\cpython\cpython\Lib\concurrent\futures\process.py", line 560 in join_executor_internals
  File "D:\a\cpython\cpython\Lib\concurrent\futures\process.py", line 514 in terminate_broken
  File "D:\a\cpython\cpython\Lib\concurrent\futures\process.py", line 344 in run
  File "D:\a\cpython\cpython\Lib\threading.py", line 1059 in _bootstrap_inner
  File "D:\a\cpython\cpython\Lib\threading.py", line 1016 in _bootstrap

Thread 0x000012cc (most recent call first):
  File "D:\a\cpython\cpython\Lib\threading.py", line 1153 in _wait_for_tstate_lock
  File "D:\a\cpython\cpython\Lib\threading.py", line 1133 in join
  File "D:\a\cpython\cpython\Lib\concurrent\futures\process.py", line 843 in shutdown
  File "D:\a\cpython\cpython\Lib\concurrent\futures\_base.py", line 647 in __exit__
  File "D:\a\cpython\cpython\Lib\test\test_concurrent_futures\test_deadlock.py", line 236 in test_crash_big_data

@vstinner vstinner merged commit a8cae40 into python:main Sep 6, 2023
@vstinner vstinner deleted the test_concurrent_futures branch September 6, 2023 13:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants