The prometheus_exporter gem is used to gather application metrics.
Currently the exporter runs in a separate container (laa-court-data-ui-metrics) on the default port 9394 within the same pod as the application container (laa-court-data-ui-app).
Metrics are scraped by Cloud Platform's Prometheus instance, and can be queried by executing a suitable promql query, e.g.
ruby_http_duration_seconds_sum{namespace="laa-court-data-ui-staging"}
There are currently 2 separate Grafana dashboards that display metrics.
One displays the status of the Kubernetes pods and is defined in kubernetes_pods_dashboard.yaml. Note the metrics displayed in this dashboard are collected by Cloud Platform independently of the Prometheus Exporter gem.
The second displays numbers of different types of http status request codes and is defined in http_status_codes_grafana_dashboard.yaml. This does display one of the metrics scraped by Prometheus Exporter.
Changes to these dashboards are applied automatically by Cloud Platform Concourse.
PrometheusRules for AlertManager alert conditions are defined in prometheus.yaml for each namespace. These are integrated with Slack using a slack webhook contained in the lcdui-secrets#SLACK_ALERTS_WEBHOOK secret.
Changes to these alerts are applied automatically by Cloud Platform Concourse.