Sidecar: Telemetry & Metrics

[merklefruit]

Bolt-sidecar telemetry

Preview Give feedback

1. Review the level of existing traces, and change or remove any unnecessary ones
2. Add the option to instrument a Prometheus listener to collect useful metrics
3. Add meaningful metrics to the sidecar, including health-checks, alerts, and important events
4. Infra: create a default Grafana dashboard to visualize the sidecar status
5. (optional) Add the option to instrument a log collector such as a remote Loki instance
6. (optional) Add the option to log to a file in JSON to support external log collectors like Promtail

[thedevbirb]

I'd love to take this, since I'm quite unfamiliar with these tools and it'd be a great way to learn them

[thedevbirb]

Since we're now taking this milestone I will approach this soon. @merklefruit wdyt if I take inspiration from Bolt RPC, regarding telemetry? I'm think about this: https://github.com/chainbound/bolt-rpc/blob/main/src/telemetry/mod.rs.

[merklefruit]

Sure, that sounds like a good starting point. For the sidecar we'd likely skip Axiom though

[namn-grg]

Sounds good! (I think its more efficient if single person does the whole issue so unassigning myself)