[wip] Updating flytekit to handle dereferencing lists of promises (local) #5
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
The goal is to be able to do the kinds of things present in
test_dataclass_elem_list_construction.py
, namely be able to take the output of a promise, wrap it in a list or other collection, then pass the result elsewhere.At the moment this breaks due to dataclass deserialization in flyte, as the derived promises don't have all the information required.
Implementation
Right now the implementation here only works for local, but gets to the core of the issue - promises need to be resolved before the
TypeEngine.to_literal
call on these goes through. WhenTypeEngine.to_literal
gets called on a collection containing adataclass
value, it's not properly dereferenced.This additionally is an issue when deserializing an int, as the fix for float -> int occurs in the dataclass transformer, which is not invoked when dereferencing, so the type is always resolved to
float
. For local, we can do a quick fix for this (adding the type information to the sub-nodes of the promise, then correcting if necessary.Expanding to remote?
No idea where to get started on this, will reach out to Flyte folks.