Tracing and diagnostics logging

[BjarkeMeier]

Synopsis

As a DataHub stakeholder
I want operations (the developers) to be able to efficiently identify the cause of system issues
So that we can quickly fix issues

"An application is properly instrumented when developers don’t need to add more instrumentation to troubleshoot an issue, because they have all of the information they need."
-- https://opentelemetry.io/docs/concepts/observability-primer/

Acceptance Criteria

• ADR-123 has been approved
• All domains adhere to ADR-123
• All logging from DataHub 3 code targets Application Insights
• All operationally connected cloud instances are also connected on the Application Map in Application Insights (for example a completed calculation in wholesale must be traceable over the Service Bus to the EDI domain)
• All teams have been introduced to how to write good tracing and diagnostics logging
• All domains have been extended with selected logging (hosted services started? use cases committed? ...? what are the general guidelines - if any?)
• All teams have been introduced to how to use Application Insights efficiently for trouble-shooting tracing and diagnostics problems
• An Azure dashboard shows the system-wide diagnostics state of DataHub 3. It provides a first-attempt solution for a single place to look to see if there are any issues that need attention from operations. It's useful to the 3rd-level support.
• Log-level can be changed in production to e.g. enable debug level for further details (how easy should this be and at what granularity level?)

Tech Notes

Remember Databricks and please note that IHostedServices doesn't hook into the ASP.NET middleware pipeline.