Handle hard failures (eg. kernel panic) during bootstrap #1143
Conversation
This change configures cloud-init to run the bootstrap script on every boot so that we can check for previous failed attempts and mark the instance as unhealthy. This caters for the scenario where the kernel panics during a bootstrap attempt which leaves the instance non-operational on reboot but marked healthy.
| return 1 | ||
| fi | ||
| } | ||
| check_docker |
There was a problem hiding this comment.
Failure here will now end up in the error trap rather than exiting immediately leaving the instance non-operational.
There was a problem hiding this comment.
Thanks for this @trvrnrth, this looks like a good improvement.
I suppose it's quite possible that one of the earlier scripts would fail too. But it looks like in those cases, the instance won't be marked as unhealthy. I do wonder if they are idempotent. It looks like if the instance is restarted, they will run again. So I think in that case, we may want to skip them in the if they completed previously. In particular, it looks like bin/bk-mount-instance-storage.sh will format the volume again on restart, which seems like something we may not want to do. You may want to do something about that, but as that's the status quo, it's no impediment to merging this.
Having said that, kernel panics causing the startup scripts to fail is not something we encounter regularly, so we haven't seen the need to make these scripts resilient to them. Do you have any insight into why they may be happening?
The issue that led to this is that we were seeing relatively frequent occurences of the kernel watchdog detecting soft lockups during installation of gcloud as part of our bootstrap. I'm unsure what the true underlying cause is. We don't currently run with instance storage enabled, but my understanding is that the boothooks will run on every boot so you're probably correct that at least some portion of them should also be skipped if they have previously completed successfully. |
This change configures cloud-init to run the bootstrap script on every boot so that we can check for previous failed attempts and mark the instance as unhealthy.
This caters for the scenario where the kernel panics during a bootstrap attempt which leaves the instance non-operational on reboot but marked healthy.