Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Moving from flytepropeller - Adding flags for ignore-retry-cause and default-max-attempts #4153

Merged
merged 16 commits into from
Oct 6, 2023
Merged

Moving from flytepropeller - Adding flags for ignore-retry-cause and default-max-attempts #4153

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
196450b
added flags for ignore-retry-cause and default-max-attempts
hamersaw Jul 26, 2023
c556d22
Merge branch 'master' into feature/simplify-retries
hamersaw Sep 14, 2023
9d253cd
Switch non interruptible @ unified retry behavior (#610)
fellhorn Sep 14, 2023
bdcb576
fixed lint issues
hamersaw Sep 14, 2023
97d44bb
began unit tests
hamersaw Sep 14, 2023
3283c29
enabling negative interruptibleFailureThreshold
hamersaw Sep 14, 2023
962c548
fixed unit tests
hamersaw Sep 14, 2023
759eaeb
Finish simplify retry behavior PR (#623)
fellhorn Sep 29, 2023
020a691
Merge branch 'master' into feature/simplify-retries
hamersaw Sep 29, 2023
b84171c
updated flyteplugins
hamersaw Sep 29, 2023
b46a2f9
Merge branch 'master' into feature/simplify-retries
hamersaw Sep 29, 2023
6116341
fixed lint issue
hamersaw Sep 29, 2023
d67e0ca
Merge branch 'master' into feature/simplify-retries
hamersaw Sep 29, 2023
4907c68
Transfer commits
eapolinario Oct 3, 2023
3a90d2d
fixed monorepo transition
hamersaw Oct 6, 2023
d767dc0
Merge branch 'master' into monorepo-move/flytepropeller/feature/simpl…
hamersaw Oct 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion flyteplugins/go/tasks/pluginmachinery/core/exec_metadata.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,6 @@ type TaskExecutionMetadata interface {
GetSecurityContext() core.SecurityContext
IsInterruptible() bool
GetPlatformResources() *v1.ResourceRequirements
GetInterruptibleFailureThreshold() uint32
GetInterruptibleFailureThreshold() int32
GetEnvironmentVariables() map[string]string
}
10 changes: 5 additions & 5 deletions flyteplugins/go/tasks/pluginmachinery/core/mocks/task_execution_metadata.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion flyteplugins/go/tasks/plugins/array/k8s/subtask_exec_context.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -268,7 +268,7 @@ func NewSubTaskExecutionMetadata(taskExecutionMetadata pluginsCore.TaskExecution
}

subTaskExecutionID := NewSubTaskExecutionID(taskExecutionMetadata.GetTaskExecutionID(), executionIndex, retryAttempt)
interruptible := taskExecutionMetadata.IsInterruptible() && uint32(systemFailures) < taskExecutionMetadata.GetInterruptibleFailureThreshold()
interruptible := taskExecutionMetadata.IsInterruptible() && int32(systemFailures) < taskExecutionMetadata.GetInterruptibleFailureThreshold()
return SubTaskExecutionMetadata{
taskExecutionMetadata,
utils.UnionMaps(taskExecutionMetadata.GetAnnotations(), secretsMap),
Expand Down
4 changes: 0 additions & 4 deletions ...est/testdata/snacks-core/k8s/002_core.containerization.multi_images.my_workflow_2_wf.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,6 @@ spec:
kind: task
name: svm_trainer
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"core.containerization.multi_images.svm_trainer" '
n1:
id: n1
Expand Down Expand Up @@ -104,8 +102,6 @@ spec:
kind: task
name: svm_predictor
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"core.containerization.multi_images.svm_predictor" '
start-node:
id: start-node
Expand Down
12 changes: 0 additions & 12 deletions ...mpiler/test/testdata/snacks-core/k8s/010_core.containerization.raw_container.wf_2_wf.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,8 +125,6 @@ spec:
kind: task
name: ellipse-area-metadata-shell
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"ellipse-area-metadata-shell" '
n1:
id: n1
Expand All @@ -144,8 +142,6 @@ spec:
kind: task
name: ellipse-area-metadata-python
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"ellipse-area-metadata-python" '
n2:
id: n2
Expand All @@ -163,8 +159,6 @@ spec:
kind: task
name: ellipse-area-metadata-r
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"ellipse-area-metadata-r" '
n3:
id: n3
Expand All @@ -182,8 +176,6 @@ spec:
kind: task
name: ellipse-area-metadata-haskell
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"ellipse-area-metadata-haskell" '
n4:
id: n4
Expand All @@ -201,8 +193,6 @@ spec:
kind: task
name: ellipse-area-metadata-julia
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"ellipse-area-metadata-julia" '
n5:
id: n5
Expand Down Expand Up @@ -260,8 +250,6 @@ spec:
kind: task
name: report_all_calculated_areas
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"core.containerization.raw_container.report_all_calculated_areas" '
start-node:
id: start-node
Expand Down
6 changes: 0 additions & 6 deletions ...stdata/snacks-core/k8s/015_core.containerization.use_secrets.my_secret_workflow_2_wf.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,24 +107,18 @@ spec:
kind: task
name: secret_task
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"core.containerization.use_secrets.secret_task" '
n1:
id: n1
kind: task
name: user_info_task
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"core.containerization.use_secrets.user_info_task" '
n2:
id: n2
kind: task
name: secret_file_task
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"core.containerization.use_secrets.secret_file_task" '
start-node:
id: start-node
Expand Down
4 changes: 0 additions & 4 deletions .../test/testdata/snacks-core/k8s/019_core.control_flow.chain_tasks.chain_tasks_wf_2_wf.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,16 +75,12 @@ spec:
kind: task
name: write
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"core.control_flow.chain_tasks.write" '
n1:
id: n1
kind: task
name: read
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"core.control_flow.chain_tasks.read" '
start-node:
id: start-node
Expand Down
6 changes: 0 additions & 6 deletions ...piler/test/testdata/snacks-core/k8s/026_core.control_flow.conditions.multiplier_2_wf.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,8 +106,6 @@ spec:
kind: branch
name: fractions
resources: {}
retry:
minAttempts: 1
n0-n0:
id: n0-n0
inputBindings:
Expand All @@ -119,8 +117,6 @@ spec:
kind: task
name: double
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"core.control_flow.conditions.double" '
n0-n1:
id: n0-n1
Expand All @@ -133,8 +129,6 @@ spec:
kind: task
name: square
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"core.control_flow.conditions.square" '
start-node:
id: start-node
Expand Down
6 changes: 0 additions & 6 deletions ...ler/test/testdata/snacks-core/k8s/028_core.control_flow.conditions.multiplier_2_2_wf.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,8 +128,6 @@ spec:
kind: branch
name: fractions
resources: {}
retry:
minAttempts: 1
n0-n0:
id: n0-n0
inputBindings:
Expand All @@ -141,8 +139,6 @@ spec:
kind: task
name: double
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"core.control_flow.conditions.double" '
n0-n1:
id: n0-n1
Expand All @@ -155,8 +151,6 @@ spec:
kind: task
name: square
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"core.control_flow.conditions.square" '
start-node:
id: start-node
Expand Down
8 changes: 0 additions & 8 deletions ...ler/test/testdata/snacks-core/k8s/030_core.control_flow.conditions.multiplier_3_2_wf.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,8 +134,6 @@ spec:
kind: branch
name: fractions
resources: {}
retry:
minAttempts: 1
n0-n0:
id: n0-n0
inputBindings:
Expand All @@ -147,8 +145,6 @@ spec:
kind: task
name: double
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"core.control_flow.conditions.double" '
n0-n1:
id: n0-n1
Expand All @@ -161,8 +157,6 @@ spec:
kind: task
name: square
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"core.control_flow.conditions.square" '
n1:
id: n1
Expand All @@ -175,8 +169,6 @@ spec:
kind: task
name: double
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"core.control_flow.conditions.double" '
start-node:
id: start-node
Expand Down
8 changes: 0 additions & 8 deletions ...test/testdata/snacks-core/k8s/035_core.control_flow.conditions.basic_boolean_wf_2_wf.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,8 +86,6 @@ spec:
kind: task
name: coin_toss
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"core.control_flow.conditions.coin_toss" '
n1:
branch:
Expand All @@ -111,23 +109,17 @@ spec:
kind: branch
name: test
resources: {}
retry:
minAttempts: 1
n1-n0:
id: n1-n0
kind: task
name: success
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"core.control_flow.conditions.success" '
n1-n1:
id: n1-n1
kind: task
name: failed
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"core.control_flow.conditions.failed" '
start-node:
id: start-node
Expand Down
6 changes: 0 additions & 6 deletions ...er/test/testdata/snacks-core/k8s/037_core.control_flow.conditions.bool_input_wf_2_wf.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,23 +91,17 @@ spec:
kind: branch
name: test
resources: {}
retry:
minAttempts: 1
n0-n0:
id: n0-n0
kind: task
name: success
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"core.control_flow.conditions.success" '
n0-n1:
id: n0-n1
kind: task
name: failed
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"core.control_flow.conditions.failed" '
start-node:
id: start-node
Expand Down
12 changes: 0 additions & 12 deletions ...est/testdata/snacks-core/k8s/039_core.control_flow.conditions.nested_conditions_2_wf.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,8 +132,6 @@ spec:
kind: branch
name: fractions
resources: {}
retry:
minAttempts: 1
n0-n0:
branch:
elseFail:
Expand Down Expand Up @@ -179,8 +177,6 @@ spec:
kind: branch
name: inner_fractions
resources: {}
retry:
minAttempts: 1
n0-n0-n0:
id: n0-n0-n0
inputBindings:
Expand All @@ -192,8 +188,6 @@ spec:
kind: task
name: double
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"core.control_flow.conditions.double" '
n0-n0-n1:
id: n0-n0-n1
Expand All @@ -206,8 +200,6 @@ spec:
kind: task
name: square
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"core.control_flow.conditions.square" '
n0-n1:
id: n0-n1
Expand All @@ -220,8 +212,6 @@ spec:
kind: task
name: square
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"core.control_flow.conditions.square" '
n0-n2:
id: n0-n2
Expand All @@ -234,8 +224,6 @@ spec:
kind: task
name: double
resources: {}
retry:
minAttempts: 1
task: 'resource_type:TASK name:"core.control_flow.conditions.double" '
start-node:
id: start-node
Expand Down
Loading
Loading