Acquire read lock when marshalling container/task. #2299
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Acquire read lock when marshalling container/task. Without acquiring the lock the agent can crash due to concurrent access.
For example, when we are modifying the fields in container/task, there can be another goroutine invoking ForceSave to save the state, which calls the marshal method, and if the marshalling doesn't acquire the read lock, the program will crash even if the other goroutine acquires the write lock before modifying the field. See Testing section for an example.
Implementation details
Wraps container and task's MarshalJSON method with read lock.
Testing
For unit test, rely on existing statemanager unit test since this PR just adds the lock.
For manual test, this isn't easily reproducible, but it can be reproduced by intentionally invoking task.PopulateSecrets many times (see this) in the code, and run a task that uses secret, and without the fix the agent will crash like the following:
I was able to verify that the issue is fixed with this commit.
Description for the changelog
Fixed a race condition that might cause the agent to crash during container creation.
Licensing
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.