Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shallow copying EnvironmentVariables map before injecting ArrayNode env vars #5182

Merged
merged 1 commit into from
Apr 4, 2024

Conversation

hamersaw
Copy link
Contributor

@hamersaw hamersaw commented Apr 4, 2024

Tracking issue

fixes #5171

Why are the changes needed?

In ArrayNode executions where the execution context has environment variables each subNode evaluation will attempt to inject maptask specific variables into the same map. This is because golang passes all maps by reference.

What changes were proposed in this pull request?

In this PR we updated the arrayExecutionContext to shallow copy existing environment variables so each subNode evaluation maintains their own replica before injecting maptask specific environment variables. This ensures that there can not be concurrent map writes in the case of parallelized subNode evaluations.

How was this patch tested?

Printed memory addresses of EnvironmentVariables map before and after change and viewed a change.

Setup process

NA

Screenshots

NA

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

NA

Docs link

NA

@dosubot dosubot bot added size:S This PR changes 10-29 lines, ignoring generated files. bug Something isn't working labels Apr 4, 2024
@hamersaw hamersaw requested a review from pvditt April 4, 2024 14:53
Copy link

codecov bot commented Apr 4, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 59.06%. Comparing base (dc1294e) to head (f8fe4ae).
Report is 10 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5182      +/-   ##
==========================================
- Coverage   59.11%   59.06%   -0.05%     
==========================================
  Files         645      646       +1     
  Lines       55574    55726     +152     
==========================================
+ Hits        32852    32916      +64     
- Misses      20129    20214      +85     
- Partials     2593     2596       +3     
Flag Coverage Δ
unittests 59.06% <100.00%> (-0.05%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Apr 4, 2024
@hamersaw hamersaw merged commit 452538a into master Apr 4, 2024
47 of 48 checks passed
@hamersaw hamersaw deleted the bug/arraynode-concurrent-map-writes branch April 4, 2024 19:20
Jeinhaus pushed a commit to Jeinhaus/flyte that referenced this pull request Apr 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working lgtm This PR has been approved by a maintainer size:S This PR changes 10-29 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] ArrayNode - fatal error: concurrent map writes
2 participants