Observability Platform using Grafana with Loki, instead of Grafana with Prometheus

[SimonPPledger]

User Story

As a Modernisation Platform Engineer or a Modernisation Platform user
I need to use Observability Platform/Tools
So that I can verify the health/cost/security/usage/capacity of the Modernisation Platform or the Application (or Application Infrastructure)

Value / Purpose

This is to investigate using using Grafana with Loki, instead of the currentl (AP) solution of using Grafana with Prometheus - ie is this a better solution?
We can compare it to other Grafana based solutions and understand advantages and disadvantages of them all.

The platform should be assessed and compared to other solutions based on the following criteria:

ease of use (this could be, but is not limited to: documentation, user friendliness/experience from both end user and engineering user point of view, configuration)
management overhead
price
scalability
expandability
limitations

Context / Background

The Modernisation Platform has a need to observe the platform and application teams have a need to observe their applications or application infrastructure.

Currently Modernisation Platform uses a number of monitoring and alerting tools/platforms: Observability Platform (by AP), slack alerting, AWS Cloud Watch -> SNS -> PagerDuty -> slack notifications, Security Hub data.
Application teams may or may not be using observability tools to meet their observability needs.

There is a need to have a centralised place where Modernisation Platform and Modernisation Platform users can observe their workloads. This PoC is one of a number of solutions we are going to explore. The solution should also take it into account that this may be the future Observability Platform for the whole MoJ.

Useful Contacts

Aaron

Additional Information

Useful documents:

State of observability in Modernisation Platform
Observability Tools research
AP Observability Platform code
AP Observability Platform tenant module

Definition of Done

#8679

#8680

#8681

#8682

#8683