-
Notifications
You must be signed in to change notification settings - Fork 672
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[House keeping] remove setting max size bytes in node context #5092
Conversation
Signed-off-by: Paul Dittamo <[email protected]>
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #5092 +/- ##
==========================================
- Coverage 58.99% 58.98% -0.02%
==========================================
Files 645 645
Lines 55648 55649 +1
==========================================
- Hits 32831 32826 -5
- Misses 20222 20228 +6
Partials 2595 2595
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
return RemoteFileOutputReader{ | ||
outPath: outPaths, | ||
store: store, | ||
maxPayloadSize: maxDatasetSize, | ||
maxPayloadSize: maxPayloadSize, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is there an issue with just removing the maxPayloadSize
completely? IIUC there is just the two checks in this file. IMO the maxDownloadMBs
applied on the data store retrieval would catch these first.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I should've mentioned this in the PR - I had some concern if there were any swaps with an external datastore implementation as mentioned in this comment. This external implementation might not have checks against maxDownloadMBs as we do in stow_store.
This is probably an unfound concern and we could just mention in release notes about potential payload size issues if you're using an external storage implementation.
Signed-off-by: Paul Dittamo <[email protected]>
Tracking issue
#4487
Why are the changes needed?
#4852 deprecated the max-output-size-bytes propeller config in favor of maxDownloadMBs storage config. This made setting MaxSizeBytes in NodeExecutionContext no longer needed as we can just read that config when reading an output file.
What changes were proposed in this pull request?
How was this patch tested?
Setup process
Screenshots
Check all the applicable boxes
Related PRs
#4852
Docs link