Adjust node filesystem space filling up warning threshold to 20% #1357
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Reduce threshold of NodeFilesystemSpaceFillingUp warning alert to 20% space available, instead of 40% (default). This will align the threshold according to default kubelet GC values below[1], "imageMinimumGCAge": "2m0s", "imageGCHighThresholdPercent": 85, "imageGCLowThresholdPercent": 80, [1] https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/ Signed-off-by: Arunprasad Rajkumar <[email protected]>
Signed-off-by: Arunprasad Rajkumar <[email protected]>
Signed-off-by: Arunprasad Rajkumar <[email protected]>
dgrisonnet
approved these changes
Sep 6, 2021
There was a problem hiding this comment.
LGTM, good job @arajkumar for finding the actual values used by the garbage collector 👍
Credit goes to @paulfantom and Simon Reber. I'm just an executor of their idea :) |
2 tasks
arajkumar
added a commit
to arajkumar/kube-prometheus
that referenced
this pull request
Apr 13, 2022
…let GC behavior Previously[1] we attempted to do the same, but there was a misunderstanding about the GC behavior and it caused the alert to be fired even before GC comes into play. According to[2][3] kubelet GC kicks in only when `imageGCHighThresholdPercent` is hit which is set to 85% by default. However `NodeFilesystemSpaceFillingUp` is set to fire as soon as 80% usage is hit. This commit changes the `fsSpaceFillingUpWarningThreshold` to 15% so that we give ample time to GC to reclaim unwanted images. This commit also changes `fsSpaceFillingUpCriticalThreshold` to 10% which gives more time to admins to react to warning before sending critical alert. [1] prometheus-operator#1357 [2] https://docs.openshift.com/container-platform/4.10/nodes/nodes/nodes-nodes-garbage-collection.html#nodes-nodes-garbage-collection-images_nodes-nodes-configuring [3] https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/ Signed-off-by: Arunprasad Rajkumar <[email protected]>
5 tasks
arajkumar
added a commit
to arajkumar/kube-prometheus
that referenced
this pull request
Apr 27, 2022
…let GC behavior Previously[1] we attempted to do the same, but there was a misunderstanding about the GC behavior and it caused the alert to be fired even before GC comes into play. According to[2][3] kubelet GC kicks in only when `imageGCHighThresholdPercent` is hit which is set to 85% by default. However `NodeFilesystemSpaceFillingUp` is set to fire as soon as 80% usage is hit. This commit changes the `fsSpaceFillingUpWarningThreshold` to 15% so that we give ample time to GC to reclaim unwanted images. This commit also changes `fsSpaceFillingUpCriticalThreshold` to 10% which gives more time to admins to react to warning before sending critical alert. [1] prometheus-operator#1357 [2] https://docs.openshift.com/container-platform/4.10/nodes/nodes/nodes-nodes-garbage-collection.html#nodes-nodes-garbage-collection-images_nodes-nodes-configuring [3] https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/ Signed-off-by: Arunprasad Rajkumar <[email protected]> (cherry picked from commit 6ff8bfb)
5 tasks
arajkumar
added a commit
to arajkumar/cluster-monitoring-operator
that referenced
this pull request
May 10, 2022
…let GC behavior Previously[1] we attempted to do the same, but there was a misunderstanding about the GC behavior and it caused the alert to be fired even before GC comes into play. According to[2][3] kubelet GC kicks in only when imageGCHighThresholdPercent is hit which is set to 85% by default. However NodeFilesystemSpaceFillingUp is set to fire as soon as 80% usage is hit. This commit changes the fsSpaceFillingUpWarningThreshold to 15% so that we give ample time to GC to reclaim unwanted images. This commit also changes fsSpaceFillingUpCriticalThreshold to 10% which gives more time to admins to react to warning before sending critical alert. [1] prometheus-operator/kube-prometheus#1357 [2] https://docs.openshift.com/container-platform/4.10/nodes/nodes/nodes-nodes-garbage-collection.html#nodes-nodes-garbage-collection-images_nodes-nodes-configuring [3] https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/ Fixes https://bugzilla.redhat.com/show_bug.cgi?id=2074807 Signed-off-by: Arunprasad Rajkumar <[email protected]>
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Related to #294
Reduce threshold of NodeFilesystemSpaceFillingUp warning alert to 20% space available, instead of 40% (default).
This will align the threshold according to default kubelet GC values
below[1],
"imageMinimumGCAge": "2m0s",
"imageGCHighThresholdPercent": 85,
"imageGCLowThresholdPercent": 80,
[1] https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/
Signed-off-by: Arunprasad Rajkumar [email protected]
Type of change
What type of changes does your code introduce to the kube-prometheus? Put an
xin the box that apply.CHANGE(fix or feature that would cause existing functionality to not work as expected)FEATURE(non-breaking change which adds functionality)BUGFIX(non-breaking change which fixes an issue)ENHANCEMENT(non-breaking change which improves existing functionality)NONE(if none of the other choices apply. Example, tooling, build system, CI, docs, etc.)Changelog entry