Using current Observability Platform (AP solution) PoC

[ewastempel]

User Story

As a Modernisation Platform Engineer or a Modernisation Platform user
I need to use Observability Platform/Tools
So that I can verify the health/cost/security/usage/capacity of the Modernisation Platform or the Application (or Application Infrastructure)

Value / Purpose

This is to understand what it means if we were to continue using the existing AP solution of the Observability platform and so that we can compare it to other Grafana based solutions and understand advantages and disadvantages of them all.

The platform should be assessed and compared to other solutions based on the following criteria:

• ease of use (this could be, but is not limited to: documentation, user friendliness/experience from both end user and engineering user point of view, configuration)
• management overhead
• price
• scalability
• expandability
• limitations

Context / Background

The Modernisation Platform has a need to observe the platform and application teams have a need to observe their applications or application infrastructure.

Currently Modernisation Platform uses a number of monitoring and alerting tools/platforms: Observability Platform (by AP), slack alerting, AWS Cloud Watch -> SNS -> PagerDuty -> slack notifications, Security Hub data.
Application teams may or may not be using observability tools to meet their observability needs.

There is a need to have a centralised place where Modernisation Platform and Modernisation Platform users can observe their workloads. This PoC is one of a number of solutions we are going to explore. The solution should also take it into account that this may be the future Observability Platform for the whole MoJ.

Useful Contacts

@ASTRobinson

Additional Information

Useful documents:

• State of observability in Modernisation Platform
• Observability Tools research
• AP Observability Platform code
• AP Observability Platform tenant module

Definition of Done

• Observability Platform - Add the modernisation-platform aws account as a cloudwatch data source #8692
• Admin role usage in the Modernisation Platform is observed in the AP Observability Platform #8679
• EC2 capacity metrics (CPU/memory/storage) are observed in the AP Observability Platform #8680
• Modernisation Platform Incidents are observed in the AP Observability Platform #8681
• Failed Modernisation Platform pipelines are observed in the AP Observability Platform #8682
• The solution is evaluated against the criteria stated in the Value/Purpose section #8683

[SimonPPledger]

will require this #8692