Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] In some cases Flyte Admin not respecting terminate call #280

Closed
4 of 20 tasks
wild-endeavor opened this issue Apr 17, 2020 · 5 comments
Closed
4 of 20 tasks

[BUG] In some cases Flyte Admin not respecting terminate call #280

wild-endeavor opened this issue Apr 17, 2020 · 5 comments
Assignees
@katrogan @anandswaminathan
Labels
bug Something isn't working
Milestone
0.17.0

Comments

@wild-endeavor
Copy link
Contributor

wild-endeavor commented Apr 17, 2020 •
edited by kumare3
Loading

Describe the bug
A user reported an issue where clicking the Terminate button in the UI did not terminate a workflow.

Expected behavior
User expected the UI to show the state of the workflow as Terminated instead of Running.

Flyte component

  • Overall
  • Flyte Setup and Installation scripts
  • Flyte Documentation
  • Flyte communication (slack/email etc)
  • FlytePropeller
  • FlyteIDL (Flyte specification language)
  • Flytekit (Python SDK)
  • FlyteAdmin (Control Plane service)
  • FlytePlugins
  • DataCatalog
  • FlyteStdlib (common libraries)
  • FlyteConsole (UI)
  • Other

To Reproduce
Have not been able to reproduce.

Screenshots
N/A

Environment
Flyte component

  • Sandbox (local or on one machine)
  • Cloud hosted
    • AWS
    • GCP
    • Azure
  • Baremetal
  • Other

Additional context
Have confirmed in logs the receipt of the TerminateExecution call, and the Aborted event that propeller sends back to Admin.
@wild-endeavor wild-endeavor added bug Something isn't working untriaged This issues has not yet been looked at by the Maintainers labels Apr 17, 2020
@kumare3
Copy link
Contributor

kumare3 commented Apr 19, 2020

I agree, either should terminated (terminating) or return an error!

@kumare3 kumare3 changed the title [BUG] Flyte Admin not respecting terminate call [BUG] In some cases Flyte Admin not respecting terminate call Apr 22, 2020
@anandswaminathan
Copy link
Contributor

+1 on Terminating state. We are seeing more issues/cases where Propeller is missing to send the event. Remember this should be a blocking Admin call from Propeller. cc @katrogan @kumare3

@kumare3 kumare3 removed the untriaged This issues has not yet been looked at by the Maintainers label Mar 1, 2021
@kumare3 kumare3 added this to the 0.13.0 milestone Mar 1, 2021
@kumare3
Copy link
Contributor

kumare3 commented May 5, 2021

I have been able to consistently reproduce this. Its for cases in which the workflows are extremely large. Flyteadmin does respect the terminate call, but it does not update the state. so UI will get updated eventually when propeller updates the state. This is especially a problem when we have system errors and propeller is backing off

@kumare3 kumare3 modified the milestones: 0.13.0, 0.14.0 May 5, 2021
@kumare3 kumare3 modified the milestones: 0.14.0, 0.15.0 Jun 6, 2021
@kumare3
Copy link
Contributor

kumare3 commented Jun 27, 2021

@katrogan / @anandswaminathan can one of you help with this issue?
cc @jeevb

@EngHabu EngHabu modified the milestones: 0.15.0, 0.16.0 Jul 7, 2021
@EngHabu EngHabu modified the milestones: 0.16.0, 0.17.0 Aug 2, 2021
@EngHabu
Copy link
Contributor

EngHabu commented Aug 31, 2021

Not enough information to go on. Please reopen if it's persistently happening... and we can jump on slack to debug it..

@EngHabu EngHabu closed this as completed Aug 31, 2021
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 6, 2022
@flyte-bot
Update Boilerplate (flyteorg#280)
8f2592f 
Signed-off-by: Flyte-Bot <[email protected]>

Co-authored-by: flyte-bot <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 20, 2022
@hamersaw
Adding primaryContainerName implementation to podBuilder (flyteorg#280)
1637021 
* Adding primaryContainerName implementation to podBuilder

Signed-off-by: Ailin Yu <[email protected]>

* Debugging: Mergo needs a pointer, and an excessive amount of debug printouts

Signed-off-by: Ailin Yu <[email protected]>

* Starting to do something, lots of debug outputs

Signed-off-by: Ailin Yu <[email protected]>

* Sidecar uses task exec ID

Signed-off-by: Ailin Yu <[email protected]>

* Cleaning up debugging

Signed-off-by: Ailin Yu <[email protected]>

* Modified container merging loop, and some dev/testing changes in sidecarbuilder

Signed-off-by: Ailin Yu <[email protected]>

* Sidecar uses primary container name from config

Signed-off-by: Ailin Yu <[email protected]>

* Cleanups

Signed-off-by: Ailin Yu <[email protected]>

* added support for default and primary container templates

Signed-off-by: Daniel Rammer <[email protected]>

* fixed container template reference issues

Signed-off-by: Daniel Rammer <[email protected]>

* removed unnecessary DeepCopy call

Signed-off-by: Daniel Rammer <[email protected]>

* added unit tests

Signed-off-by: Daniel Rammer <[email protected]>

* fixed lint issues

Signed-off-by: Daniel Rammer <[email protected]>

Signed-off-by: Ailin Yu <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Co-authored-by: Daniel Rammer <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 20, 2022
@kumare3
Nested conditions example (flyteorg#280)
6951858 
Signed-off-by: Ketan Umare <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 20, 2022
@pmahindrakar-oss
Fixed validation logic for launch plan schedule registration (flyteor…
e45aabf 
…g#280)
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 20, 2022
@pmahindrakar-oss
Added env flag for sandbox start (flyteorg#273)
936134e 
* Added env flag for sandbox start

Signed-off-by: Prafulla Mahindrakar <[email protected]>

* ADDED docs for FLYTE_TIMEOUT

Signed-off-by: Prafulla Mahindrakar <[email protected]>

* Fixed validation logic for launch plan schedule registration (flyteorg#280)

Signed-off-by: Prafulla Mahindrakar <[email protected]>

Signed-off-by: pmahindrakar-oss <[email protected]>

* boilerplate updates

Signed-off-by: Prafulla Mahindrakar <[email protected]>

* fixed unit test after stdlib upgrade

Signed-off-by: Prafulla Mahindrakar <[email protected]>

* fixed unit test

Signed-off-by: Prafulla Mahindrakar <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 9, 2023
@flyte-bot
Update Boilerplate (flyteorg#280)
3d8e597 
Signed-off-by: Flyte-Bot <[email protected]>

Co-authored-by: flyte-bot <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 21, 2023
@hamersaw
Adding primaryContainerName implementation to podBuilder (flyteorg#280)
b790ed3 
* Adding primaryContainerName implementation to podBuilder

Signed-off-by: Ailin Yu <[email protected]>

* Debugging: Mergo needs a pointer, and an excessive amount of debug printouts

Signed-off-by: Ailin Yu <[email protected]>

* Starting to do something, lots of debug outputs

Signed-off-by: Ailin Yu <[email protected]>

* Sidecar uses task exec ID

Signed-off-by: Ailin Yu <[email protected]>

* Cleaning up debugging

Signed-off-by: Ailin Yu <[email protected]>

* Modified container merging loop, and some dev/testing changes in sidecarbuilder

Signed-off-by: Ailin Yu <[email protected]>

* Sidecar uses primary container name from config

Signed-off-by: Ailin Yu <[email protected]>

* Cleanups

Signed-off-by: Ailin Yu <[email protected]>

* added support for default and primary container templates

Signed-off-by: Daniel Rammer <[email protected]>

* fixed container template reference issues

Signed-off-by: Daniel Rammer <[email protected]>

* removed unnecessary DeepCopy call

Signed-off-by: Daniel Rammer <[email protected]>

* added unit tests

Signed-off-by: Daniel Rammer <[email protected]>

* fixed lint issues

Signed-off-by: Daniel Rammer <[email protected]>

Signed-off-by: Ailin Yu <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Co-authored-by: Daniel Rammer <[email protected]>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Apr 30, 2024
@pmahindrakar-oss
Fixed validation logic for launch plan schedule registration (flyteor…
9cb07f9 
…g#280)
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Apr 30, 2024
@pmahindrakar-oss
Added env flag for sandbox start (flyteorg#273)
39add71 
* Added env flag for sandbox start

Signed-off-by: Prafulla Mahindrakar <[email protected]>

* ADDED docs for FLYTE_TIMEOUT

Signed-off-by: Prafulla Mahindrakar <[email protected]>

* Fixed validation logic for launch plan schedule registration (flyteorg#280)

Signed-off-by: Prafulla Mahindrakar <[email protected]>

Signed-off-by: pmahindrakar-oss <[email protected]>

* boilerplate updates

Signed-off-by: Prafulla Mahindrakar <[email protected]>

* fixed unit test after stdlib upgrade

Signed-off-by: Prafulla Mahindrakar <[email protected]>

* fixed unit test

Signed-off-by: Prafulla Mahindrakar <[email protected]>
austin362667 pushed a commit to austin362667/flyte that referenced this issue May 7, 2024
@pmahindrakar-oss @austin362667
Fixed validation logic for launch plan schedule registration (flyteor…
bcbd668 
…g#280)
austin362667 pushed a commit to austin362667/flyte that referenced this issue May 7, 2024
@pmahindrakar-oss @austin362667
Added env flag for sandbox start (flyteorg#273)
a60953d 
* Added env flag for sandbox start

Signed-off-by: Prafulla Mahindrakar <[email protected]>

* ADDED docs for FLYTE_TIMEOUT

Signed-off-by: Prafulla Mahindrakar <[email protected]>

* Fixed validation logic for launch plan schedule registration (flyteorg#280)

Signed-off-by: Prafulla Mahindrakar <[email protected]>

Signed-off-by: pmahindrakar-oss <[email protected]>

* boilerplate updates

Signed-off-by: Prafulla Mahindrakar <[email protected]>

* fixed unit test after stdlib upgrade

Signed-off-by: Prafulla Mahindrakar <[email protected]>

* fixed unit test

Signed-off-by: Prafulla Mahindrakar <[email protected]>
robert-ulbrich-mercedes-benz pushed a commit to robert-ulbrich-mercedes-benz/flyte that referenced this issue Jul 2, 2024
@pmahindrakar-oss @robert-ulbrich-mercedes-benz
Fixed validation logic for launch plan schedule registration (flyteor…
913b161 
…g#280)
robert-ulbrich-mercedes-benz pushed a commit to robert-ulbrich-mercedes-benz/flyte that referenced this issue Jul 2, 2024
@pmahindrakar-oss @robert-ulbrich-mercedes-benz
Added env flag for sandbox start (flyteorg#273)
6dc3992 
* Added env flag for sandbox start

Signed-off-by: Prafulla Mahindrakar <[email protected]>

* ADDED docs for FLYTE_TIMEOUT

Signed-off-by: Prafulla Mahindrakar <[email protected]>

* Fixed validation logic for launch plan schedule registration (flyteorg#280)

Signed-off-by: Prafulla Mahindrakar <[email protected]>

Signed-off-by: pmahindrakar-oss <[email protected]>

* boilerplate updates

Signed-off-by: Prafulla Mahindrakar <[email protected]>

* fixed unit test after stdlib upgrade

Signed-off-by: Prafulla Mahindrakar <[email protected]>

* fixed unit test

Signed-off-by: Prafulla Mahindrakar <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@katrogan katrogan

@anandswaminathan anandswaminathan

Labels
bug Something isn't working
Projects
None yet
Milestone
0.17.0
Development

No branches or pull requests
5 participants
@EngHabu @katrogan @wild-endeavor @kumare3 @anandswaminathan