[BUG] Retry fetching subworkflow output data on failure

[pvditt]

Tracking issue

Closes: #4369

Why are the changes needed?

• Errors are not bubbled up when fetching a subworkflow's output fails. This causes for failed output fetches to be handled later on when reading nil output literals leading to unclear/confusing error messages.

- When remote store reads fail in GetOutputs or GetInputs, errors are not bubbled up - returning the URL blob to the client instead. However, the URL blob is deprecated. Sending a clear error status improves clarity. Update: moving this to another PR

• GetLimitMegabytes could potentially differ between propeller and admin given certain deployment configuration potentially causing for the GetExecutionData call to fetch subworklow output data could always fail while succeed if called on propeller. There is ~similar logic utilized in recoverInputs.

What changes were proposed in this pull request?

• Retry fetching subworkflow output data on propeller when GetExecutionData admin call fails.
  - Bubble up errors when fetching Input and Output for workflow, node and task executions instead of returning empty values. Update: moving this to another PR
  - Remove UrlBlob utilization when fetching Input and Output for workflow, node and task execution Update: moving this to another PR

How was this patch tested?

• Added/updated unit tests
• Ran subworkflows.py parent_workflow in flytesnack to see if GetExecutionData + manually set GetExecutionData to fail to test if the remote store retry in propeller would work.

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

[Future-Outlier]

Amazing PR, I am bothered by this problem ...
I will test this.

[codecov]

Codecov Report

Attention: 9 lines in your changes are missing coverage. Please review.

Comparison is base (ac42562) 58.13% compared to head (1f161f9) 58.21%.
Report is 7 commits behind head on master.

Files	Patch %	Lines
flytepropeller/pkg/controller/controller.go	0.00%	9 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4602      +/-   ##
==========================================
+ Coverage   58.13%   58.21%   +0.07%     
==========================================
  Files         626      626              
  Lines       53786    53796      +10     
==========================================
+ Hits        31271    31316      +45     
+ Misses      20007    19972      -35     
  Partials     2508     2508              

Flag	Coverage Δ
unittests	58.21% <72.72%> (+0.07%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pvditt]

@hamersaw I removed the FlyteAdmin changes from this PR. Will open up a house keeping PR with those changes.