[3.8] bpo-39360: Ensure all workers exit when finalizing a multiprocessing Pool (GH-19009) #19023
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…Pool (pythonGH-19009) When the pull is not used via the context manager or terminate() is called, there is a system in multiprocessing.util that handles finalization of all pools via an atexit handler (the Finalize) class. This class registers the _terminate_pool handler in the registry of finalizers of the module, and that registry is called on interpreter exit via _exit_function. The problem is that the "happy" path with the context manager or manual call to finalize() does some extra steps that _terminate_pool does not. The step that is not executed when the atexit() handler calls _terminate_pool is pinging the _change_notifier queue to unblock the maintenance threads. This commit moves the notification to the _terminate_pool function so is called from both code paths. Co-authored-by: Pablo Galindo <[email protected]> (cherry picked from commit ac10e0c) Co-authored-by: Batuhan Taşkaya <[email protected]>
|
@isidentical and @pablogsal: Status check is done, and it's a success ✅ . |
2 similar comments
|
@isidentical and @pablogsal: Status check is done, and it's a success ✅ . |
|
@isidentical and @pablogsal: Status check is done, and it's a success ✅ . |
pablogsal
approved these changes
Mar 15, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When the pull is not used via the context manager or terminate() is called, there is a system in multiprocessing.util that handles finalization of all pools via an atexit handler (the Finalize) class. This class registers the _terminate_pool handler in the registry of finalizers of the module, and that registry is called on interpreter exit via _exit_function. The problem is that the "happy" path with the context manager or manual call to finalize() does some extra steps that _terminate_pool does not. The step that is not executed when the atexit() handler calls _terminate_pool is pinging the _change_notifier queue to unblock the maintenance threads.
This commit moves the notification to the _terminate_pool function so is called from both code paths.
Co-authored-by: Pablo Galindo [email protected]
(cherry picked from commit ac10e0c)
Co-authored-by: Batuhan Taşkaya [email protected]
https://bugs.python.org/issue39360