Metrics for Relayer

[jackzampolin]

Ideally when running rly start the relayer should expose a Prometheus metrics server which exposes metrics about the operations. This should include:

• Counter of messages by type
• Counter of gas used
• Counter of fees paid by denom/address

This issue is also a place for other ideas for metrics. Please add them as comments!

[wangfeiping]

I think client update time is also important so that relayer operators can customize alarms.
for example:
client_update_status{chain_id="dawnsworld-1b",time="2020-05-21T07:04:29.507285347Z"} 100
Value represents client duration (seconds).

[colin-axner]

reference on how SDK recently added metrics: cosmos/cosmos-sdk#6449

[DavidNix]

Are there any other metrics besides these?

• Counter of messages by type
• Counter of gas used
• Counter of fees paid by denom/address

As to the last one Counter of fees paid by denom/address, we may not want to capture address.

My assumption is we will use prometheus-style metrics. Denom and address sounds like it's an unbounded sets of labels.

Per prometheus best practices: https://prometheus.io/docs/practices/naming/#labels

CAUTION: Remember that every unique combination of key-value label pairs represents a new time series, which can dramatically increase the amount of data stored. Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values.

We may be able to get buy with denom but I recommend not adding addresses as a label.

[agouin]

Duplicate of #877