Add metrics proxying to external collectors.

[divan]

While the main purpose of statusmonitor is to give real-time metrics for the debug sessions, it might be useful to forward collected data to the external metrics collection services like Grafana, influxdb, statsd, Datadog etc.

Currently, there is an option to dump data into CSV file for further analysis, and this logic can be extended and generalized to support pretty much any type of "exporting".

What needs to be done:

• choose the external collector that is most convenient to people who interested in collecting metrics for tests/debug runs.
• implement the actual export code
• add a configuration flags (command line probably, to keep it simple)

[dshulyak]

Is there any reason why we don't want to export metrics directly from statusd node? I am talking about using prometheus, and it is not possible to push metrics in this case. Basically, it requires an HTTP handler, similar to expvar, which will be polled periodically. It is also possible to use expvar and convert data into prometheus format, but some prometheus related metadata will be lost in this case:

https://github.com/prometheus/collectd_exporter/blob/master/vendor/github.com/prometheus/client_golang/prometheus/expvar.go

As I understood there are some metrics that need to be collected externally, but usually people are writing small daemons for that purpose.

[divan]

@dshulyak good question and we have many discussions on that, but the general idea is that we want a lot of different metrics, but only for the debug builds/test sessions. Release builds that are shipped to mobile phones should not send anything to any centralized server.

While there are different use cases (see status-im/swarms#51 for example), the main purpose of statusmonitor is to give real-time zero-configuration insights into the performance related metrics of the running app. The typical case is: developer tries to reproduce a bug or performance issue on the own device and watching metrics using statusmonitor without spending the time to configuring it, setting up infrastructure, etc.

Idea of extending its functionality and providing proxying data to the external collector is something that people keep wondering about and so it makes sense to add.

For cases like "Testing cluster" where we definitely want a lot of instances to report a lot of metrics, probably a separate daemon should be used. We don't have test cluster yet, so it's up to use how to design the whole thing.

Also, make sure to check:
status-im/status-go#523
status-im/status-go#421