Configurable minimum memory for all workloads

[mwarkentin]

Which component are you using?:

VerticalPodAutoscaler

Is your feature request designed to solve a problem? If so describe the problem this feature should solve.:

We recently enabled VPA on some workloads - one of the containers (pgbouncer) has extremely low memory usage (1-3Mi).

VPA set memory request and limit to ~3Mi as expected, however our containers started to crash, and were not resolved by VPAs OOM handling.

We saw errors like the following: Error: failed to start containerd task "pgbouncer": cannot start a stopped process: unknown

I am wondering if the containers running on GKE require a certain minimum amount of memory in order to start at all, based on some of these references to similar errors that I found:

• Low memory limit causes OCI runtime create failed when running container moby/moby#38921
• [CVE-2019-5736]: Runc uses more memory during start up after the fix opencontainers/runc#1980
• low memory on start causing strange failure condition moby/moby#41164

Describe the solution you'd like.:

If that's the case, it would be nice if I could enforce a hard minimum (maybe 10Mi) across all of my VPAs, which container resources would not be set below, even if the container policy set a minimum of 1Mi

Describe any alternative solutions you've considered.:

Right now, my approach will be to require setting a container policy for every container, and trying to enforce that they always have a value of 10Mi or higher everywhere.

Additional context.:

[kaiiyvwu]

I think you can add a extra arg --pod-recommendation-min-memory-mb={Int} on vpa-recommender args

[adrianmoisey]

/area vertical-pod-autoscaler

[adrianmoisey]

That is correct, you can set a minimum:
https://github.com/kubernetes/autoscaler/blob/vertical-pod-autoscaler-1.2.1/vertical-pod-autoscaler/pkg/recommender/logic/recommender.go#L30

Interestingly, the default is 250MiB, which is strange that you got such a low value.

/assign

[mwarkentin]

I'm using the GKE VPA, so I'm guessing this is something I'll need to take up with support there.

Thanks for the pointers, if you'd like this can be closed.

[adrianmoisey]

I see that their docs say that there is no default minimum: https://cloud.google.com/kubernetes-engine/docs/concepts/verticalpodautoscaler#containerresourcepolicy_v1_autoscalingk8sio

[mwarkentin]

Thanks, I think that confirms this is an issue with GKE VPA only!