Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] A failed wf node leaves the other nodes (spark tasks) running until they finish #263

Closed
3 of 20 tasks
EngHabu opened this issue Apr 13, 2020 · 1 comment
Closed
3 of 20 tasks

[BUG] A failed wf node leaves the other nodes (spark tasks) running until they finish #263

EngHabu opened this issue Apr 13, 2020 · 1 comment
Labels
bug Something isn't working untriaged This issues has not yet been looked at by the Maintainers

Comments

@EngHabu
Copy link
Contributor

EngHabu commented Apr 13, 2020

Describe the bug
If a node in a workflow fails, the entire workflow fails, if there are other nodes running spark, the underlying CRD will continue to run wasting resources.

Expected behavior
If the workflow shows as failed, the underlying CRDs it created should be cleaned up.

Flyte component

  • Overall
  • Flyte Setup and Installation scripts
  • Flyte Documentation
  • Flyte communication (slack/email etc)
  • FlytePropeller
  • FlyteIDL (Flyte specification language)
  • Flytekit (Python SDK)
  • FlyteAdmin (Control Plane service)
  • FlytePlugins
  • DataCatalog
  • FlyteStdlib (common libraries)
  • FlyteConsole (UI)
  • Other

Environment
Flyte component

  • Sandbox (local or on one machine)
  • Cloud hosted
    • AWS
    • GCP
    • Azure
  • Baremetal
  • Other
@EngHabu EngHabu added bug Something isn't working untriaged This issues has not yet been looked at by the Maintainers labels Apr 13, 2020
@kumare3
Copy link
Contributor

kumare3 commented Apr 22, 2020

@EngHabu does the abort not propagate?

@EngHabu EngHabu mentioned this issue Apr 22, 2020
Merged
8 tasks
@EngHabu EngHabu closed this as completed Apr 22, 2020
@EngHabu EngHabu mentioned this issue Apr 30, 2020
Merged
8 tasks
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 6, 2022
@regadas
Fix: register remaining tasks without default (flyteorg#263)
f48fdff 
* Fix: register remaining tasks without default plugins

Signed-off-by: Filipe Regadas <[email protected]>

* Add test

Signed-off-by: Filipe Regadas <[email protected]>

* fixup! Add test

Signed-off-by: Filipe Regadas <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 6, 2022
@EngHabu
Migrate to golang-jwt/jwt/v4 (flyteorg#263)
6df5cbf 
* Migrate to golang-jwt/jwt/v4

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* go mod tidy

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Move to go 1.17

Signed-off-by: Haytham Abuelfutuh <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 20, 2022
@yindia
feat: added reuseable workflow for boilerplate (flyteorg#263)
82be16b 
Signed-off-by: Yuvraj <[email protected]>

Co-authored-by: Yuvraj <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 20, 2022
@SandraGH5
Update README.rst (flyteorg#263)
1c73954
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 20, 2022
@flyte-bot @pmahindrakar-oss
Update documentation (flyteorg#263)
fdeeef2 
Signed-off-by: Flyte-Bot <[email protected]>

Co-authored-by: pmahindrakar-oss <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 9, 2023
@regadas
Fix: register remaining tasks without default (flyteorg#263)
d8a4a6e 
* Fix: register remaining tasks without default plugins

Signed-off-by: Filipe Regadas <[email protected]>

* Add test

Signed-off-by: Filipe Regadas <[email protected]>

* fixup! Add test

Signed-off-by: Filipe Regadas <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 21, 2023
@EngHabu
Migrate to golang-jwt/jwt/v4 (flyteorg#263)
818843b 
* Migrate to golang-jwt/jwt/v4

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* go mod tidy

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Move to go 1.17

Signed-off-by: Haytham Abuelfutuh <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 21, 2023
@yindia
feat: added reuseable workflow for boilerplate (flyteorg#263)
6dc5f39 
Signed-off-by: Yuvraj <[email protected]>

Co-authored-by: Yuvraj <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Apr 30, 2024
@flyte-bot @pmahindrakar-oss
Update documentation (flyteorg#263)
1784a38 
Signed-off-by: Flyte-Bot <[email protected]>

Co-authored-by: pmahindrakar-oss <[email protected]>
austin362667 pushed a commit to austin362667/flyte that referenced this issue May 7, 2024
@flyte-bot @pmahindrakar-oss @austin362667
Update documentation (flyteorg#263)
1da3ee0 
Signed-off-by: Flyte-Bot <[email protected]>

Co-authored-by: pmahindrakar-oss <[email protected]>
robert-ulbrich-mercedes-benz pushed a commit to robert-ulbrich-mercedes-benz/flyte that referenced this issue Jul 2, 2024
@flyte-bot @pmahindrakar-oss @robert-ulbrich-mercedes-benz
Update documentation (flyteorg#263)
e19a1eb 
Signed-off-by: Flyte-Bot <[email protected]>

Co-authored-by: pmahindrakar-oss <[email protected]>
troychiu pushed a commit that referenced this issue Jul 8, 2024
@hamersaw
gracefully abort fasttask on worker timeout (#263)
897f549 
## Overview
This PR enables graceful aborts (rather than panics) when a fasttask times out waiting for worker availability.

## Test Plan
Tested locally.

## Rollout Plan (if applicable)
This can be rolled out along with any other changes.

## Upstream Changes
Should this change be upstreamed to OSS (flyteorg/flyte)? If so, please check this box for auditing. Note, this is the responsibility of each developer. See [this guide](https://unionai.atlassian.net/wiki/spaces/ENG/pages/447610883/Flyte+-+Union+Cloud+Development+Runbook/#When-are-versions-updated%3F).
- [ ] To be upstreamed

## Jira Issue
https://unionai.atlassian.net/browse/EXO-103

## Checklist
* [ ] Added tests
* [ ] Ran a deploy dry run and shared the terraform plan
* [ ] Added logging and metrics
* [ ] Updated [dashboards](https://unionai.grafana.net/dashboards) and [alerts](https://unionai.grafana.net/alerting/list)
* [ ] Updated documentation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working untriaged This issues has not yet been looked at by the Maintainers
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@EngHabu @kumare3