Provide a way to enable Cilium/Hubble metrics

[mmkay]

Summary

Provide a way to enable Cilium/Hubble monitoring-related features via cluster-config annotations. An equivalent Helm command to enable them in the Cilium chart would be: https://docs.cilium.io/en/stable/observability/grafana/#deploy-cilium-and-hubble-with-metrics-enabled

Why is this important?

The k8s snap is currently deploying Cilium with Hubble enabled, however its metrics and exemplars are disabled. Enabling metrics and exemplars could provide a significant improvement in observability of the L7 network flows which we could build upon in Observability and expose the network flow metrics and exemplars in Grafana as well which could be useful for the cluster admins.

Are you interested in contributing to this feature?

We can help if needed, but we don't yet have the full understanding of all your inner workings :)