You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm very new to Flyte and have a question regarding accessing data within workflows.
Below is my structure for example: @workflow
def wf_example(dataset: Dict[str, pd.DataFrame]):
res1 = task1(dataset['data1'])
res2 = task2(dataset['data2'])
...
The above won't work apparently because dataset is Promise type within wf_example. Could I get some suggestions on how to properly handle situations like this?
One naive approach I'm thinking about is to have a pre_task that breaks down my Dict into NamedTuples and then my tasks can access slices of the whole dataset.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Good afternoon Flyte community,
I'm very new to Flyte and have a question regarding accessing data within workflows.
Below is my structure for example:
@workflow
def wf_example(dataset: Dict[str, pd.DataFrame]):
res1 = task1(dataset['data1'])
res2 = task2(dataset['data2'])
...
The above won't work apparently because dataset is Promise type within wf_example. Could I get some suggestions on how to properly handle situations like this?
One naive approach I'm thinking about is to have a pre_task that breaks down my Dict into NamedTuples and then my tasks can access slices of the whole dataset.
Many thanks.
Beta Was this translation helpful? Give feedback.
All reactions