Tessera /metrics endpoint does not match Prometheus format

[tsujp]

I am not an expert on Prometheus or Telegraf.

I've been attempting to read in the metrics exposed by Tessera at the endpoint /metrics using the follow Telegraf filter:

[[inputs.prometheus]]
  name_override = "tessera"
  urls = [
    # this is a running, healthy tessera container
    "http://localhost:9000/metrics"
  ]
  # this has been tried with both metric_version 2 and 1
  metric_version = 2
  response_timeout = "5s"

This results in the following error:

[inputs.prometheus] Error in plugin: error reading metrics for http://localhost:9000/metrics: reading text format failed: text format parsing error in line 280: unexpected end of input stream

I had a quick peek at the output via curl, while also counting line numbers:

curl localhost:9000/metrics | less -N

The last two lines are:

    279 tessera_THIRD_PARTY_GET_getVersion_RequestRate_requestsPerSeconds 0.0
    280 tessera_THIRD_PARTY_GET_getVersion_RequestCount 0

If you run the curl without piping into less you'll notice the return does not end in a newline. I thought this was strange and decided to test a local echo-server with two metrics, one ending in a newline and one without. I used this Javascript:

const express = require('express')

const app = express()

app.get('/metrics', (req, res) => {
	res.status(200).send('tessera_P2P_GET_getPartyInfo_MaxTime_ms 0')
})

app.get('/metrics-fixed', (req, res) => {
	res.status(200).send('tessera_P2P_GET_getPartyInfo_MaxTime_ms 0\n')
})

app.listen(9999, '0.0.0.0')

The Telegraf filter now correctly processes the /metrics-fixed endpoint and fails as expected via the /metrics endpoint. This leads me to believe Tessera is missing a required newline at the end of the response to it's /metrics endpoint.

The spec for Prometheus formatting also states newline endings here: https://prometheus.io/docs/instrumenting/exposition_formats/

I don't do Java but somewhere here https://github.com/ConsenSys/tessera/blob/47ecb17dadd1e2b8e649c9de7408959c03cdeb8a/server/jersey-server/src/main/java/com/quorum/tessera/server/monitoring/MetricsResource.java#L42 a newline is being dropped even though it's present in the PrometheusProtocolFormatter and on line 37 of the linked code too.

[macfarla]

@tsujp I've put up a PR here #1320 - are you able to build from source and test this with your setup?