Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix signal handling for non-leader processes #1680

Merged
merged 1 commit into from
Mar 17, 2023

Conversation

antonipp
Copy link
Contributor

Description

This PR fixes a small issue: when Leader Election is enabled, all non-leader processes are waiting on:
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/c261df66a00635509f7d8cb2a7fba4c602c9228e/main.go#L222

However, this means that they are never getting to this signal-handling logic below:
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/c261df66a00635509f7d8cb2a7fba4c602c9228e/main.go#L234-L238

This results in non-leader processes not responding to SIGTERM or SIGINT, those signals are simply ignored. The only way to terminate those processes is a SIGKILL.

This PR makes sure that non-leader processes can be terminated with SIGTERM or SIGINT, just like the leader process.

Testing

Verified this behaviour in our Kubernetes environment:

  • Before the PR, kill <PID> was ignored for non-leader processes but worked for the leader process
  • After the PR, kill <PID> successfully terminates both leader and non-leader processes

@google-cla
Copy link

google-cla bot commented Feb 14, 2023

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Copy link
Collaborator

@liyinan926 liyinan926 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks.

@liyinan926 liyinan926 merged commit bc05d55 into kubeflow:master Mar 17, 2023
@antonipp antonipp deleted the fix-signal-handling branch March 22, 2023 13:27
ImpSy pushed a commit to spotinst/spark-on-k8s-operator that referenced this pull request Jul 27, 2023
ImpSy pushed a commit to spotinst/spark-on-k8s-operator that referenced this pull request Jul 31, 2023
jbhalodia-slack pushed a commit to jbhalodia-slack/spark-operator that referenced this pull request Oct 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants