bootstrapper: prioritize etcd disk I/O #3114
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for constellation-docs canceled.
|
burgerdev
reviewed
May 21, 2024
msanft
force-pushed
the
feat/bootstrapper/etcd-io-prio
branch
from
11d3bcc to
af411ae
Compare
daniel-weisse
approved these changes
May 22, 2024
|
Lets get this merged so we can see if it fixes things during our daily and weekly e2e tests |
burgerdev
reviewed
May 22, 2024
burgerdev
approved these changes
May 22, 2024
msanft
force-pushed
the
feat/bootstrapper/etcd-io-prio
branch
from
e5e8166 to
a1f883f
Compare
Coverage report
|
daniel-weisse
approved these changes
May 22, 2024
burgerdev
approved these changes
May 22, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context
The etcd issues like the notorious "etcdserver: leader changed" are most likely caused due to etcd's disk activity conflicting with other disk-heavy operations during cluster initialization / joining such as the pulling of container images. To mitigate this, we can prioritize the disk operations of the etcd process over those of other processes by giving it an I/O priority in the kernel. It will not consider pod restarts of etcd, which are out of scope for this PR.
How to move forward:
We'll need to see (over a course of, say, 2 weeks) if this change is enough to get rid of the issues. If not, we'll probably need to use faster disks per default on Azure. This would probably require switching to the (~90€/M) 512Gb P20 disk, which satisfies the requirements of etcd. This should be done in a separate testing period to gather differential results.
Proposed change(s)
Additional info
Checklist