Array node map task #1640
Merged
Array node map task #1640
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: eduardo apolinario <[email protected]>
Signed-off-by: eduardo apolinario <[email protected]>
eapolinario
requested review from
wild-endeavor,
kumare3,
pingsutw and
cosmicBboy
as code owners
Signed-off-by: eduardo apolinario <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: eduardo apolinario <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: eduardo apolinario <[email protected]>
Signed-off-by: eduardo apolinario <[email protected]>
Signed-off-by: eduardo apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: eduardo apolinario <[email protected]>
Signed-off-by: eduardo apolinario <[email protected]>
Signed-off-by: eduardo apolinario <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: eduardo apolinario <[email protected]>
Signed-off-by: eduardo apolinario <[email protected]>
Signed-off-by: eduardo apolinario <[email protected]>
Signed-off-by: eduardo apolinario <[email protected]>
Signed-off-by: eduardo apolinario <[email protected]>
Signed-off-by: eduardo apolinario <[email protected]>
wild-endeavor
approved these changes
Aug 4, 2023
fg91
pushed a commit
that referenced
this pull request
Aug 15, 2023
* Test experimental map task so far Signed-off-by: eduardo apolinario <[email protected]> * Lint Signed-off-by: eduardo apolinario <[email protected]> * Include ArrayNodeMapTask task spec in list of registrable entities Signed-off-by: eduardo apolinario <[email protected]> * can start fast-register Signed-off-by: Daniel Rammer <[email protected]> * fixed task type Signed-off-by: Daniel Rammer <[email protected]> * Add _execute_map_task Signed-off-by: eduardo apolinario <[email protected]> * added self to _compute_array_job_index function Signed-off-by: Daniel Rammer <[email protected]> * overridding _outputs_interface Signed-off-by: Daniel Rammer <[email protected]> * imports Signed-off-by: Daniel Rammer <[email protected]> * added get_type_for_output_var override Signed-off-by: Daniel Rammer <[email protected]> * using correct python outputs type Signed-off-by: Daniel Rammer <[email protected]> * not appending task index to output_prefix for ArrayNode Signed-off-by: Daniel Rammer <[email protected]> * using metadata from subtasks Signed-off-by: Daniel Rammer <[email protected]> * updated task interface Signed-off-by: Daniel Rammer <[email protected]> * reverted interface - will need to handle in propeller compiler Signed-off-by: Daniel Rammer <[email protected]> * Hash metadata * Fix bindings to ArrayNode Signed-off-by: eduardo apolinario <[email protected]> * Minor cleanup Signed-off-by: eduardo apolinario <[email protected]> * Remove extraneous code Signed-off-by: eduardo apolinario <[email protected]> * Use correct version of flyteidl Signed-off-by: eduardo apolinario <[email protected]> * More cleanup Signed-off-by: eduardo apolinario <[email protected]> * Use experimental module Signed-off-by: eduardo apolinario <[email protected]> * Fix tests Signed-off-by: eduardo apolinario <[email protected]> * Add support for partials Signed-off-by: eduardo apolinario <[email protected]> * Add support for fast registration Signed-off-by: eduardo apolinario <[email protected]> * Linting Signed-off-by: eduardo apolinario <[email protected]> * Remove a few TODOs Signed-off-by: eduardo apolinario <[email protected]> --------- Signed-off-by: eduardo apolinario <[email protected]> Signed-off-by: Daniel Rammer <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: eduardo apolinario <[email protected]> Co-authored-by: Daniel Rammer <[email protected]>
Future-Outlier
pushed a commit
to Future-Outlier/flytekit
that referenced
this pull request
Oct 3, 2023
* Test experimental map task so far Signed-off-by: eduardo apolinario <[email protected]> * Lint Signed-off-by: eduardo apolinario <[email protected]> * Include ArrayNodeMapTask task spec in list of registrable entities Signed-off-by: eduardo apolinario <[email protected]> * can start fast-register Signed-off-by: Daniel Rammer <[email protected]> * fixed task type Signed-off-by: Daniel Rammer <[email protected]> * Add _execute_map_task Signed-off-by: eduardo apolinario <[email protected]> * added self to _compute_array_job_index function Signed-off-by: Daniel Rammer <[email protected]> * overridding _outputs_interface Signed-off-by: Daniel Rammer <[email protected]> * imports Signed-off-by: Daniel Rammer <[email protected]> * added get_type_for_output_var override Signed-off-by: Daniel Rammer <[email protected]> * using correct python outputs type Signed-off-by: Daniel Rammer <[email protected]> * not appending task index to output_prefix for ArrayNode Signed-off-by: Daniel Rammer <[email protected]> * using metadata from subtasks Signed-off-by: Daniel Rammer <[email protected]> * updated task interface Signed-off-by: Daniel Rammer <[email protected]> * reverted interface - will need to handle in propeller compiler Signed-off-by: Daniel Rammer <[email protected]> * Hash metadata * Fix bindings to ArrayNode Signed-off-by: eduardo apolinario <[email protected]> * Minor cleanup Signed-off-by: eduardo apolinario <[email protected]> * Remove extraneous code Signed-off-by: eduardo apolinario <[email protected]> * Use correct version of flyteidl Signed-off-by: eduardo apolinario <[email protected]> * More cleanup Signed-off-by: eduardo apolinario <[email protected]> * Use experimental module Signed-off-by: eduardo apolinario <[email protected]> * Fix tests Signed-off-by: eduardo apolinario <[email protected]> * Add support for partials Signed-off-by: eduardo apolinario <[email protected]> * Add support for fast registration Signed-off-by: eduardo apolinario <[email protected]> * Linting Signed-off-by: eduardo apolinario <[email protected]> * Remove a few TODOs Signed-off-by: eduardo apolinario <[email protected]> --------- Signed-off-by: eduardo apolinario <[email protected]> Signed-off-by: Daniel Rammer <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: eduardo apolinario <[email protected]> Co-authored-by: Daniel Rammer <[email protected]> Signed-off-by: Future Outlier <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TL;DR
This PR adds support for map tasks defined in terms of ArrayNode
TODO
map_task(a_mappable_task)andmap_task(a_mappable_task, concurrency=2)may both be used at the same timeType
Are all requirements met?
Complete description
Extremely WIP at the moment.
Need to verify support for container tasks and other backend entities.
Tracking Issue
flyteorg/flyte#1131
Follow-up issue
NA
OR
https://github.com/flyteorg/flyte/issues/