Optimizing Kubernetes with Kwok: Replay Peak Load with Prometheus Metrics?

[topikachu]

Hi,
Kwok is a powerful tool that can help us optimize Kubernetes resource configurations, allowing us to manage our system and schedule pods more efficiently. Currently, we're leveraging it to enhance our system pod schedule, but we're looking to take things a step further.

We'd like to simulate runtime behavior by replaying resource consumption data based on historical metrics from Prometheus. This way, we can test if our new configurations handle peak load effectively.

Any suggestions or recommendations on setting up and executing this simulation would be greatly appreciated!

[wzshiming]

Currently, kwok has only been tested with metrics-server for resource usage, but if you're using Prometheus you may need to adapt its configuration to capture resource usage information based on the paths in the Node annotation.

https://kwok.sigs.k8s.io/docs/user/resource-usage-configuration/#out-of-the-box

[topikachu]

@wzshiming
Thank you for the suggestion! The simulation you described uses fixed usage values from pod annotations or CEL expressions. I’m interested in dynamically reporting values from a remote feed instead of using static data. For example, could we implement a CEL function like QueryRemote("/some/remote/feed") to pull these values?

[wzshiming]

It seems just needs to have a controller that changes the value of ResourceUsage at regular intervals.

https://kwok.sigs.k8s.io/docs/user/resource-usage-configuration/