Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rollback when upgrade process fails #153

Closed
adrianmo opened this issue Apr 1, 2019 · 1 comment · Fixed by #255
Closed

Rollback when upgrade process fails #153

adrianmo opened this issue Apr 1, 2019 · 1 comment · Fixed by #255
Assignees
@pbelgundi
Labels
area/controller Issue related to the operator controller area/upgrade Impacts upgrade feature in operator kind/feature New feature priority/P1 Recoverable error, functionality/performance impaired but not lost, no permanent damage status/blocked Issue or PR is blocked on another item; add reference in a comment version 0.4.0 Issue with Operator 0.4.0

Comments

@adrianmo
Copy link
Contributor

adrianmo commented Apr 1, 2019

When the version field is updated on the PravegaCluster resource, the operator triggers a rolling upgrade process that updates BookKeeper, Segmentstore, and Controller pods until all of them are using the desired version.

During the upgrade process, various errors can happen, from a version that does not exists to a failure in the configuration that prevents the pod from becoming ready.

At this moment, when a failure occurs, the upgrade process is halted and requires manual intervention to restore the cluster's health.

The operator should attempt to automatically restore the cluster health, rolling back to the original version. If the rollback fails, the process should halt and an alert should be sent.
@adrianmo adrianmo added kind/feature New feature area/controller Issue related to the operator controller labels Apr 1, 2019
@adrianmo
Copy link
Contributor Author

Blocked by #157.

@adrianmo adrianmo added priority/P1 Recoverable error, functionality/performance impaired but not lost, no permanent damage status/blocked Issue or PR is blocked on another item; add reference in a comment labels May 30, 2019
@pbelgundi pbelgundi added the version 0.4.0 Issue with Operator 0.4.0 label Jul 2, 2019
@pbelgundi pbelgundi added the area/upgrade Impacts upgrade feature in operator label Jul 25, 2019
@Tristan1900 Tristan1900 mentioned this issue Aug 27, 2019
Closed
@pbelgundi pbelgundi mentioned this issue Sep 4, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pbelgundi pbelgundi

Labels
area/controller Issue related to the operator controller area/upgrade Impacts upgrade feature in operator kind/feature New feature priority/P1 Recoverable error, functionality/performance impaired but not lost, no permanent damage status/blocked Issue or PR is blocked on another item; add reference in a comment version 0.4.0 Issue with Operator 0.4.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Issue 153: Add Rollback support for Pravega Cluster pravega/pravega-operator
2 participants
@adrianmo @pbelgundi