This repository has been archived by the owner on Oct 23, 2023. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 61
Support union and none type in flyteidl #401
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Yubo Wang <[email protected]>
Signed-off-by: Yubo Wang <[email protected]>
yubofredwang
force-pushed
the
support_union_scalar
branch
from
May 8, 2023 20:29
6b6208f
to
e3c7da5
Compare
Signed-off-by: Yubo Wang <[email protected]>
Codecov Report
@@ Coverage Diff @@
## master #401 +/- ##
==========================================
+ Coverage 76.11% 78.62% +2.51%
==========================================
Files 18 18
Lines 1390 1240 -150
==========================================
- Hits 1058 975 -83
+ Misses 280 209 -71
- Partials 52 56 +4
Flags with carried forward coverage won't be shown. Click here to find out more.
|
hamersaw
approved these changes
May 9, 2023
pingsutw
approved these changes
May 9, 2023
eapolinario
pushed a commit
that referenced
this pull request
May 16, 2023
* add support for Union Scalar Signed-off-by: Yubo Wang <[email protected]> * support union type and literals Signed-off-by: Yubo Wang <[email protected]> * change union type extraction Signed-off-by: Yubo Wang <[email protected]> --------- Signed-off-by: Yubo Wang <[email protected]> Co-authored-by: Yubo Wang <[email protected]> Co-authored-by: Kevin Su <[email protected]>
eapolinario
added a commit
that referenced
this pull request
May 16, 2023
This reverts commit 3284f61. Signed-off-by: Eduardo Apolinario <[email protected]>
eapolinario
added a commit
that referenced
this pull request
May 16, 2023
* added dynamic_job_spec_uri to dynamic workflow metadata and node execution closure (#360) Signed-off-by: Daniel Rammer <[email protected]> * Use TokenCache in ClientCredentialsTokenSourceProvider (#377) * Init customTokenSource.refreshTime (#381) Signed-off-by: Andrew Dye <[email protected]> * added DataLoadingConfig to K8sPod message (#368) Signed-off-by: Daniel Rammer <[email protected]> * Add Reasons field to TaskExecutionClosure to track time-series of reasons (#382) * added a time-series of reasons to the TaskExecution closure Signed-off-by: Daniel Rammer <[email protected]> * added docs Signed-off-by: Daniel Rammer <[email protected]> * actually finishing docs too Signed-off-by: Daniel Rammer <[email protected]> --------- Signed-off-by: Daniel Rammer <[email protected]> * Create service for runtime metrics (#367) * added span messages Signed-off-by: Daniel Rammer <[email protected]> * added endpoints to service Signed-off-by: Daniel Rammer <[email protected]> * generated mocks Signed-off-by: Daniel Rammer <[email protected]> * removed get task execution metrics rpc Signed-off-by: Daniel Rammer <[email protected]> * added EXECUTION_IDLE category Signed-off-by: Daniel Rammer <[email protected]> * updated PLUGIN_EXECUTION to PLUGIN_RUNTIME Signed-off-by: Daniel Rammer <[email protected]> * removed recorded_at on workflow and node level events Signed-off-by: Daniel Rammer <[email protected]> * added docs for task event reported_at field Signed-off-by: Daniel Rammer <[email protected]> * removed GetNodeExecutionMetrics endpoint - will implement later if necessary Signed-off-by: Daniel Rammer <[email protected]> * updated docs Signed-off-by: Daniel Rammer <[email protected]> * added reported_at for node execution events Signed-off-by: Daniel Rammer <[email protected]> * fixed typo Signed-off-by: Daniel Rammer <[email protected]> * fixed typos and removed dead code Signed-off-by: Daniel Rammer <[email protected]> * updated categories Signed-off-by: Daniel Rammer <[email protected]> * added workflow setup and teardown categories Signed-off-by: Daniel Rammer <[email protected]> * simplified span message and moved to flyteidl.core Signed-off-by: Daniel Rammer <[email protected]> --------- Signed-off-by: Daniel Rammer <[email protected]> * Remove misleading token refresh logic from client credentials token source provider (#383) * Out of core plugin (#378) * Add backend plugin system service Signed-off-by: Kevin Su <[email protected]> * Add backend plugin system service Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * update state Signed-off-by: Kevin Su <[email protected]> * update state Signed-off-by: Kevin Su <[email protected]> * dics Signed-off-by: Kevin Su <[email protected]> * Remove output prefix from get request Signed-off-by: Kevin Su <[email protected]> * update Signed-off-by: Kevin Su <[email protected]> * remove prev state Signed-off-by: Kevin Su <[email protected]> * update proto Signed-off-by: Kevin Su <[email protected]> * remove error message Signed-off-by: Kevin Su <[email protected]> * update comment Signed-off-by: Kevin Su <[email protected]> * make generate Signed-off-by: Kevin Su <[email protected]> * Rename the service Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Feat: Add `ElasticConfig` message type for torch elastic training (#394) * Add elastic config args to pytorch proto Signed-off-by: Fabio Graetz <[email protected]> * Add elastic config message type for torchrun training Signed-off-by: Fabio Graetz <[email protected]> --------- Signed-off-by: Fabio Graetz <[email protected]> Co-authored-by: Fabio Grätz <[email protected]> Co-authored-by: Ketan Umare <[email protected]> * Retract 1.4.x (#397) Signed-off-by: eduardo apolinario <[email protected]> Co-authored-by: eduardo apolinario <[email protected]> * Data addresses #minor (#391) Signed-off-by: Yee Hing Tong <[email protected]> * Refactor kf-operator plugins configs and support setting different specs for different replica groups (#386) * refactor kubeflow operators proto Signed-off-by: Yubo Wang <[email protected]> * add back the original proto for backward compatible Signed-off-by: Yubo Wang <[email protected]> * clean up comments Signed-off-by: Yubo Wang <[email protected]> * add kubeflow.rs Signed-off-by: Yubo Wang <[email protected]> * add elastic config Signed-off-by: Yubo Wang <[email protected]> * add command to MPI Signed-off-by: Yubo Wang <[email protected]> * add slots and command to mpi spec Signed-off-by: Yubo Wang <[email protected]> --------- Signed-off-by: Yubo Wang <[email protected]> Co-authored-by: Yubo Wang <[email protected]> * add user_identifier (#388) Signed-off-by: byhsu <[email protected]> Signed-off-by: eduardo apolinario <[email protected]> Co-authored-by: byhsu <[email protected]> Co-authored-by: eduardo apolinario <[email protected]> * Add envs to execution spec (#400) Signed-off-by: Kevin Su <[email protected]> * Support union and none type in flyteidl (#401) * add support for Union Scalar Signed-off-by: Yubo Wang <[email protected]> * support union type and literals Signed-off-by: Yubo Wang <[email protected]> * change union type extraction Signed-off-by: Yubo Wang <[email protected]> --------- Signed-off-by: Yubo Wang <[email protected]> Co-authored-by: Yubo Wang <[email protected]> Co-authored-by: Kevin Su <[email protected]> * Rename user_identity to execution_identity (#402) Signed-off-by: byhsu <[email protected]> Co-authored-by: byhsu <[email protected]> * make generate Signed-off-by: eduardo apolinario <[email protected]> * Revert "Support union and none type in flyteidl (#401)" This reverts commit 3284f61. Signed-off-by: Eduardo Apolinario <[email protected]> * We should not update flyteidl version in backend components in the case of this branch Signed-off-by: eduardo apolinario <[email protected]> --------- Signed-off-by: Daniel Rammer <[email protected]> Signed-off-by: Andrew Dye <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Fabio Graetz <[email protected]> Signed-off-by: eduardo apolinario <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Yubo Wang <[email protected]> Signed-off-by: byhsu <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Dan Rammer <[email protected]> Co-authored-by: Andrew Dye <[email protected]> Co-authored-by: Kevin Su <[email protected]> Co-authored-by: Fabio M. Graetz, Ph.D <[email protected]> Co-authored-by: Fabio Grätz <[email protected]> Co-authored-by: Ketan Umare <[email protected]> Co-authored-by: eduardo apolinario <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Co-authored-by: Yubo Wang <[email protected]> Co-authored-by: Yubo Wang <[email protected]> Co-authored-by: ByronHsu <[email protected]> Co-authored-by: byhsu <[email protected]>
eapolinario
pushed a commit
that referenced
this pull request
Jun 27, 2023
* add support for Union Scalar Signed-off-by: Yubo Wang <[email protected]> * support union type and literals Signed-off-by: Yubo Wang <[email protected]> * change union type extraction Signed-off-by: Yubo Wang <[email protected]> --------- Signed-off-by: Yubo Wang <[email protected]> Co-authored-by: Yubo Wang <[email protected]> Co-authored-by: Kevin Su <[email protected]>
eapolinario
added a commit
that referenced
this pull request
Jun 28, 2023
* Adding support for structured dataset (#369) Signed-off-by: pmahindrakar-oss <[email protected]> * added dynamic_job_spec_uri to dynamic workflow metadata and node execution closure (#360) Signed-off-by: Daniel Rammer <[email protected]> * Use TokenCache in ClientCredentialsTokenSourceProvider (#377) * Init customTokenSource.refreshTime (#381) Signed-off-by: Andrew Dye <[email protected]> * added DataLoadingConfig to K8sPod message (#368) Signed-off-by: Daniel Rammer <[email protected]> * Add Reasons field to TaskExecutionClosure to track time-series of reasons (#382) * added a time-series of reasons to the TaskExecution closure Signed-off-by: Daniel Rammer <[email protected]> * added docs Signed-off-by: Daniel Rammer <[email protected]> * actually finishing docs too Signed-off-by: Daniel Rammer <[email protected]> --------- Signed-off-by: Daniel Rammer <[email protected]> * Create service for runtime metrics (#367) * added span messages Signed-off-by: Daniel Rammer <[email protected]> * added endpoints to service Signed-off-by: Daniel Rammer <[email protected]> * generated mocks Signed-off-by: Daniel Rammer <[email protected]> * removed get task execution metrics rpc Signed-off-by: Daniel Rammer <[email protected]> * added EXECUTION_IDLE category Signed-off-by: Daniel Rammer <[email protected]> * updated PLUGIN_EXECUTION to PLUGIN_RUNTIME Signed-off-by: Daniel Rammer <[email protected]> * removed recorded_at on workflow and node level events Signed-off-by: Daniel Rammer <[email protected]> * added docs for task event reported_at field Signed-off-by: Daniel Rammer <[email protected]> * removed GetNodeExecutionMetrics endpoint - will implement later if necessary Signed-off-by: Daniel Rammer <[email protected]> * updated docs Signed-off-by: Daniel Rammer <[email protected]> * added reported_at for node execution events Signed-off-by: Daniel Rammer <[email protected]> * fixed typo Signed-off-by: Daniel Rammer <[email protected]> * fixed typos and removed dead code Signed-off-by: Daniel Rammer <[email protected]> * updated categories Signed-off-by: Daniel Rammer <[email protected]> * added workflow setup and teardown categories Signed-off-by: Daniel Rammer <[email protected]> * simplified span message and moved to flyteidl.core Signed-off-by: Daniel Rammer <[email protected]> --------- Signed-off-by: Daniel Rammer <[email protected]> * Remove misleading token refresh logic from client credentials token source provider (#383) * Out of core plugin (#378) * Add backend plugin system service Signed-off-by: Kevin Su <[email protected]> * Add backend plugin system service Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * update state Signed-off-by: Kevin Su <[email protected]> * update state Signed-off-by: Kevin Su <[email protected]> * dics Signed-off-by: Kevin Su <[email protected]> * Remove output prefix from get request Signed-off-by: Kevin Su <[email protected]> * update Signed-off-by: Kevin Su <[email protected]> * remove prev state Signed-off-by: Kevin Su <[email protected]> * update proto Signed-off-by: Kevin Su <[email protected]> * remove error message Signed-off-by: Kevin Su <[email protected]> * update comment Signed-off-by: Kevin Su <[email protected]> * make generate Signed-off-by: Kevin Su <[email protected]> * Rename the service Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Feat: Add `ElasticConfig` message type for torch elastic training (#394) * Add elastic config args to pytorch proto Signed-off-by: Fabio Graetz <[email protected]> * Add elastic config message type for torchrun training Signed-off-by: Fabio Graetz <[email protected]> --------- Signed-off-by: Fabio Graetz <[email protected]> Co-authored-by: Fabio Grätz <[email protected]> Co-authored-by: Ketan Umare <[email protected]> * Retract 1.4.x (#397) Signed-off-by: eduardo apolinario <[email protected]> Co-authored-by: eduardo apolinario <[email protected]> * Data addresses #minor (#391) Signed-off-by: Yee Hing Tong <[email protected]> * Refactor kf-operator plugins configs and support setting different specs for different replica groups (#386) * refactor kubeflow operators proto Signed-off-by: Yubo Wang <[email protected]> * add back the original proto for backward compatible Signed-off-by: Yubo Wang <[email protected]> * clean up comments Signed-off-by: Yubo Wang <[email protected]> * add kubeflow.rs Signed-off-by: Yubo Wang <[email protected]> * add elastic config Signed-off-by: Yubo Wang <[email protected]> * add command to MPI Signed-off-by: Yubo Wang <[email protected]> * add slots and command to mpi spec Signed-off-by: Yubo Wang <[email protected]> --------- Signed-off-by: Yubo Wang <[email protected]> Co-authored-by: Yubo Wang <[email protected]> * add user_identifier (#388) Signed-off-by: byhsu <[email protected]> Signed-off-by: eduardo apolinario <[email protected]> Co-authored-by: byhsu <[email protected]> Co-authored-by: eduardo apolinario <[email protected]> * Add envs to execution spec (#400) Signed-off-by: Kevin Su <[email protected]> * Support union and none type in flyteidl (#401) * add support for Union Scalar Signed-off-by: Yubo Wang <[email protected]> * support union type and literals Signed-off-by: Yubo Wang <[email protected]> * change union type extraction Signed-off-by: Yubo Wang <[email protected]> --------- Signed-off-by: Yubo Wang <[email protected]> Co-authored-by: Yubo Wang <[email protected]> Co-authored-by: Kevin Su <[email protected]> * Rename user_identity to execution_identity (#402) Signed-off-by: byhsu <[email protected]> Co-authored-by: byhsu <[email protected]> * Single literal in GetDataResponse (#404) Signed-off-by: Yee Hing Tong <[email protected]> * Add namespace to execution system metadata (#406) Signed-off-by: Katrina Rogan <[email protected]> * Add oauth2 http proxy client (#405) Signed-off-by: byhsu <[email protected]> * Rename externalPluginService to AgentService (#410) * Rename externalPluginService to AgentService Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Add external_plugin_service proto back to the idl (#413) * Add external-plugin-service proto back to the idl Signed-off-by: Kevin Su <[email protected]> * update idl Signed-off-by: Kevin Su <[email protected]> * update idll Signed-off-by: Kevin Su <[email protected]> * update idll Signed-off-by: Kevin Su <[email protected]> * AsyncAgentService Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Rerun make generate Signed-off-by: eduardo apolinario <[email protected]> --------- Signed-off-by: pmahindrakar-oss <[email protected]> Signed-off-by: Daniel Rammer <[email protected]> Signed-off-by: Andrew Dye <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Fabio Graetz <[email protected]> Signed-off-by: eduardo apolinario <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Yubo Wang <[email protected]> Signed-off-by: byhsu <[email protected]> Signed-off-by: Katrina Rogan <[email protected]> Co-authored-by: pmahindrakar-oss <[email protected]> Co-authored-by: Dan Rammer <[email protected]> Co-authored-by: Andrew Dye <[email protected]> Co-authored-by: Kevin Su <[email protected]> Co-authored-by: Fabio M. Graetz, Ph.D <[email protected]> Co-authored-by: Fabio Grätz <[email protected]> Co-authored-by: Ketan Umare <[email protected]> Co-authored-by: eduardo apolinario <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Co-authored-by: Yubo Wang <[email protected]> Co-authored-by: Yubo Wang <[email protected]> Co-authored-by: ByronHsu <[email protected]> Co-authored-by: byhsu <[email protected]> Co-authored-by: Katrina Rogan <[email protected]>
eapolinario
pushed a commit
that referenced
this pull request
Sep 8, 2023
* add support for Union Scalar Signed-off-by: Yubo Wang <[email protected]> * support union type and literals Signed-off-by: Yubo Wang <[email protected]> * change union type extraction Signed-off-by: Yubo Wang <[email protected]> --------- Signed-off-by: Yubo Wang <[email protected]> Co-authored-by: Yubo Wang <[email protected]> Co-authored-by: Kevin Su <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TL;DR
Union type is not supported in flyteidl leading to flytectl failures when the task takes Union type as input or output.
When users do operations such as:
flytectl get execution
will output:
failed to convert to a known Literal. Input Type [union_type:..] not supported
Type
Are all requirements met?
Complete description
For creating making literal for union type, the code will find the first one that matches the input value. If the input is 1.0
and union type is Union(Integer, Float), integer will be used since 1.0 will match with integer.
For default generation of a union type, I just use the first variants possible. For example, if the union contains
Union(String, Integer), then default value generated when creating an execFile will be "" not 0.
Tested with flytectl commands, get task and create execution is working fine
Tracking Issue
fixes flyteorg/flyte#3094