Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use child dir for branch taken #6120

Merged

Conversation

andrewwdye
Copy link
Contributor

Why are the changes needed?

We reuse the same data directory for the branch node and the branch taken (child).

What changes were proposed in this pull request?

This change adds an additional set of directories to decouple these. It makes sure to copy the child output up to the branch node output like we do in subworkflow

How was this patch tested?

Example

from dataclasses import dataclass
from flytekit import conditional, task, workflow

@dataclass
class MyData:
    foo: str

@workflow
def root_wf(data: MyData) -> str:
    return sub_wf(data=data)

@workflow
def sub_wf(data: MyData) -> str:
    check = always_true()
    return conditional("decision").if_(check.is_true()).then(conditional_wf(data=data)).else_().fail("not done")

@task
def always_true() -> bool:
    return True

@workflow
def conditional_wf(data: MyData) -> str:
    return done(data)

@task
def done(data: MyData) -> str:
    return f"done ({data.foo})"
  • Add unittests
  • Run locally using sandbox and verify behavior with working example. Inspect paths

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

@andrewwdye andrewwdye requested a review from hamersaw December 20, 2024 18:38
We reuse the same data directory for the branch node and the branch taken (child). This change adds an additional set of directories to decouple these. It makes sure to copy the child output up to the branch node output like we do in subworkflow

Example
```
from dataclasses import dataclass
from flytekit import conditional, task, workflow

@DataClass
class MyData:
    foo: str

@workflow
def root_wf(data: MyData) -> str:
    return sub_wf(data=data)

@workflow
def sub_wf(data: MyData) -> str:
    check = always_true()
    return conditional("decision").if_(check.is_true()).then(conditional_wf(data=data)).else_().fail("not done")

@task
def always_true() -> bool:
    return True

@workflow
def conditional_wf(data: MyData) -> str:
    return done(data)

@task
def done(data: MyData) -> str:
    return f"done ({data.foo})"
```

- [x] Add unittests
- [x] Run locally using sandbox and verify behavior with working example. Inspect paths

Signed-off-by: Andrew Dye <[email protected]>
@andrewwdye andrewwdye force-pushed the union/upstream-f2cc8c0a709d427a9de9aac6555946d86f70c2cf branch from 2132483 to 97e17ab Compare December 20, 2024 18:41
Copy link

codecov bot commented Dec 20, 2024

Codecov Report

Attention: Patch coverage is 40.00000% with 12 lines in your changes missing coverage. Please review.

Project coverage is 36.97%. Comparing base (27b2f3a) to head (97e17ab).
Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
...tepropeller/pkg/controller/nodes/branch/handler.go 40.00% 9 Missing and 3 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6120      +/-   ##
==========================================
- Coverage   36.97%   36.97%   -0.01%     
==========================================
  Files        1318     1318              
  Lines      132506   132516      +10     
==========================================
+ Hits        48997    49000       +3     
- Misses      79254    79258       +4     
- Partials     4255     4258       +3     
Flag Coverage Δ
unittests-datacatalog 51.58% <ø> (ø)
unittests-flyteadmin 54.05% <ø> (ø)
unittests-flytecopilot 30.99% <ø> (ø)
unittests-flytectl 62.29% <ø> (ø)
unittests-flyteidl 7.23% <ø> (ø)
unittests-flyteplugins 53.85% <ø> (ø)
unittests-flytepropeller 42.59% <40.00%> (-0.01%) ⬇️
unittests-flytestdlib 55.18% <ø> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@andrewwdye andrewwdye merged commit 4ac3019 into master Dec 20, 2024
52 checks passed
@andrewwdye andrewwdye deleted the union/upstream-f2cc8c0a709d427a9de9aac6555946d86f70c2cf branch December 20, 2024 19:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants