Adding worker autoscaling support with KEDA

[sdaberdaku]

No description provided.

[sdaberdaku]

For some reason, the trino_execution_resourcegroups_InternalResourceGroup_RunningQueries JMX metric is not exposed by Trino 446. It is by versions 435 and by 467. My KEDA test starts with 0 worker replicas. Then, a query on TPCH is launched on the coordinator, which increases this value from 0 to 1 and triggers the creation of a worker pod. Unfortunately, queries that are submitted when no workers are available enter a state of "WAITING_FOR_RESOUCES" which is not blocked nor queued.

@nineinchnick I am open to suggestions for a better metric or testing strategy.

[sdaberdaku]

I found the trino_execution_ClusterSizeMonitor_RequiredWorkers metric that works with all versions of Trino and allows us to test scaling up from 0 workers.

I am also wondering if the Chart should support the creation of TriggerAuthentication objects to cover the cases when Prometheus requires authentication. One could always create this object outside the chart and reference it in the ScaledObject trigger, so it is not mandatory. What is blocking me is how to implement the trigger - TriggerAuthentication assignment. One TriggerAuthentication can be referenced by multiple triggers. Naming also needs to be deterministic, since the trigger needs to reference the TriggerAuthentication object by name. I could "inject" the {{ Release.Name }} prefix to these objects, but I don't like it very much.

[nineinchnick]

First pass, I haven't yet checked all the new properties.

[nineinchnick]

Are there any other popular autoscalers for Kubernetes? I see this just creates the ScaledObject resource. What's the value in including this in the Trino chart? How hard is to manage this as an add-on, for example using an umbrella chart?

[sdaberdaku]

Are there any other popular autoscalers for Kubernetes?

I am not aware of other open-source Kubernetes autoscalers for Pods/Jobs.

I see this just creates the ScaledObject resource. What's the value in including this in the Trino chart?

KEDA allows for scaling worker Pods down to zero replicas when the cluster is unused, and to me this is a pretty interesting feature that cannot be achieved with vanilla HPA.

How hard is to manage this as an add-on, for example using an umbrella chart?

It is definitely doable, I am currently managing it with Kustomize and ArgoCD. One pain point in having to do this outside of the Chart is that I cannot remove the spec.replicas field in the worker deployment and keep the HPA disabled (as discussed here: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#replicas, this field should not be set when using horizontal scaling). Moreover, I have to explicitly configure ArgoCD so that it ignores this particular field so that it does not try to reconcile it.

[nineinchnick]

I added the breaking-change label, because installations which were relying on server.workers=0 now also need to set coordinator.config.nodeScheduler.includeCoordinator=true.