fix(k8sprocessor): store only necessary Pod data #593
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
swiatekm
force-pushed
the
fix/k8sprocessor/cache-optimization
branch
from
d55e0d5 to
5917908
Compare
github-actions
bot
added
the
documentation
kasia-kujawa
approved these changes
Jun 1, 2022
swiatekm
force-pushed
the
fix/k8sprocessor/cache-optimization
branch
from
5917908 to
0b604a4
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Informers are a convenient mechanism for maintaining an eventually-consistent local cache of Kubernetes objects. One of the problems with them used to be that it was difficult to customize the object data stored in the cache, leading to unnecessarily large memory usage even when the client only cared about a small subset of the data.
client-go0.23.0 added a mechanism to do this, where it's possible to provide the informer with a transform function applied to the object before it's stored and event handlers are fired.We supply a transform function to only retain the Pod data we use in our cache. Truthfully, we could get rid of all of it, as we keep the computed metadata in a separate data structure, but that would require implementing a custom Store. The transform function approach is much simpler in comparison.
The result is a close to 10x reduction in processor memory usage in large clusters (5000+ Pods).