Question/Feature request: Monitoring/Alerting for Flows/Outputs

[sebastiangaiser]

Is your feature request related to a problem? Please describe.
Hey, I was thinking about how to monitor whether a Flow/Output resource is having problems or is working fine.

Describe the solution you'd like
I would like to build a kind of alerting (via PrometheusRule) indicating the status of a Flow/Output. The alerts should be sent to each team/namespace individually. For me the question is how is it supposed to work.
Monitoring and Alerting docs are not tackling this, yet.

Describe alternatives you've considered
Other tools like Flux achieved this by using kube-state-metrics (ksm). Ksm aggregates the status object and builds a Prometheus metric on top of that.

Additional context
Flux documentation: https://fluxcd.io/flux/monitoring/custom-metrics/
Flux example repository: https://github.com/fluxcd/flux2-monitoring-example

[sebastiangaiser]

@pepov sorry to ping you but could you please mind have a look :D

[pepov]

Hey, sorry for the late response! Using kube-state-metrics could be a good idea since the flow/output resources already have a status with an active flag and problem count that could be used to detect issues. What would an integration with ksm mean for the logging operator? Would that be just documentation or would that imply some automation in the operator?

[sebastiangaiser]

Hey, thank you for your response. For me an idea would be to include the ksm as optional subchart and insert a config via Helm values.

[pepov]

@sebastiangaiser in case of the prometheus operator that is a reasonable thing to have but for the logging operator this would be an overkill in my opinion. Even in case of flux the example is a demonstration of how to configure ksm through the kube-prometheus-stack to have monitoring for custom resources. I would prefer this type of support in our case as well.

[sebastiangaiser]

The flux example with the kube-prometheus-stack uses the ksm chart itself as a subchart.
For me personally, I don't like to mix the kube-prometheus-stack with any not related CRs monitoring. So for Flux, I deployed the ksm chart on my own. The downside of that is that you end up with multiple ksm deployments but this gives me independents of my logging-stack from my monitoring-stack. But of course, documenting it works out, too.

[sebastiangaiser]

Not stale

[sebastiangaiser]

@pepov can you please pin this issue