Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Workflow fetch and sub-workflows #75

Merged
merged 37 commits into from
Mar 9, 2020
Merged

Workflow fetch and sub-workflows #75

merged 37 commits into from
Mar 9, 2020

Conversation

wild-endeavor
Copy link
Contributor

@wild-endeavor wild-endeavor commented Jan 13, 2020

Please see flyteorg/flyte#139 for additional context.

  • Completes the SdkWorkflow.promote_from_model functionality. Promoting from model now optionally takes subworkflows and tasks. If specified, the ones provided will be used instead of fetching from Admin.
  • Enables the sub-workflow behavior
  • Updates flyteidl to a version that has sub-workflows in the workflow creation request to Admin.
  • Calling an SdkWorkflow now produces a node.
  • SdkTaskNode and SdkWorkflowNode have been moved into a separate file from nodes.py.

Related things but don't have time to put into this PR:

  • Proper serialization of components (we want to use the same construct as the current registration call path)
  • How do you not register subworkflows, but do register them if they're also standalone workflows, and always use the correct name, and also always have the correct dependency structure when doing the topologic sort.
  • Discovered that we have duplicate CompiledTask models while writing this PR but will fix later. Left a todo.

Yee Hing Tong added 2 commits January 9, 2020 18:57
@wild-endeavor wild-endeavor changed the title Workflow fetch Workflow fetch and sub-workflows Feb 7, 2020
@matthewphsmith matthewphsmith self-requested a review March 9, 2020 19:14
matthewphsmith
matthewphsmith previously approved these changes Mar 9, 2020
@wild-endeavor wild-endeavor merged commit fd17913 into master Mar 9, 2020
max-hoffman pushed a commit to dolthub/flytekit that referenced this pull request May 11, 2021
Please see flyteorg/flyte#139 for additional context.

* Completes the SdkWorkflow.promote_from_model functionality.  Promoting from model now optionally takes subworkflows and tasks.  If specified, the ones provided will be used instead of fetching from Admin.
* Enables the sub-workflow behavior
* Updates flyteidl to a version that has sub-workflows in the workflow creation request to Admin.
* Calling an SdkWorkflow now produces a node.
* `SdkTaskNode` and `SdkWorkflowNode` have been moved into a separate file from nodes.py.

Related things but don't have time to put into this PR:
* Proper serialization of components (we want to use the same construct as the current registration call path)
* How do you not register subworkflows, but do register them if they're also standalone workflows, and always use the correct name, and also always have the correct dependency structure when doing the topologic sort.
* Discovered that we have duplicate `CompiledTask` models while writing this PR but will fix later.  Left a todo.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants