fix synchronization in collective DistArray
initializations/transformations
#484
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There are several closely-related issues around synchronization in initialization/transformation of
DistArray
's:diagonal_array
usestaskq.fence()
which can lead to deadlock without global synchronization (e.g. if rank receives tasks not generated bydiagonal_array
whose submission/execution requires progress beyonddiagonal_array
)ntask_created == ntask_completed
(e.g. https://github.com/ValeevGroup/tiledarray/blob/master/src/TiledArray/conversions/foreach.h#L286 , withntask_created
namedtask_count
andntask_completed
namedcounter
). As is now obvious updatingntask_completed
from within the task, before its result has been assigned to the future creates a potential race, since the awaited condition will be satisfied before the effects of the task completion have been completed. The correct way to ensure local completion is to updatentask_completed
using a callback attached to the future that will be assigned the task result.To address these issues, beyond direct fixes, it would be convenient to add fencing support to collectives like
DistArray::init_tiles
directly. This PR accomplishes that.