Emit metrics via otel instead of custom format

[andreasohlund]

This replaces the homegrown metrics in the audit instance with OTEL to allow for easier consumption in visualization tools like Prometheus, Azure Monitor etc

[andreasohlund]

@lailabougria can you spot anything off in terms of how we emit the telemetry? naming etc

[ramonsmits]

@andreasohlund update your Rider formatting so that this will not be put on a single line

[ramonsmits]

This was returned in a search:

The environment variable OTEL_EXPORTER_OTLP_ENDPOINT exists as a standardized way to specify the OpenTelemetry OTLP endpoint. By default, it points to http://localhost:4317 for gRPC and http://localhost:4318 for HTTP/protobuf.
There are also some related environment variables:

OTEL_EXPORTER_OTLP_PROTOCOL for specifying the protocol (grpc, http/protobuf)
OTEL_EXPORTER_OTLP_HEADERS for additional headers
OTEL_EXPORTER_OTLP_TIMEOUT for request timeouts

Maybe instead we use a more verbose name?

OTEL_EXPORTER_OTLP_ENDPOINT

If not maybe immediately we should ensure that environment variable expansion is active so you could do:

<add key="ServiceControl.Audit/OTEL_EXPORTER_OTLP_ENDPOINT" value="%OTEL_EXPORTER_OTLP_ENDPOINT%" />

or for that matter:

SERVICECONTROL_AUDIT_OTEL_EXPORTER_OTLP_ENDPOINT=%OTEL_EXPORTER_OTLP_ENDPOINT%

[ramonsmits]

ok, while testing envvar expansion behavior I stumbled upon the settings reader:

ServiceControl/src/ServiceControl.Configuration/EnvironmentVariableSettingsReader.cs

Lines 37 to 52 in ea0cb32

	// The 3 app namespaces can be loaded from env vars without the namespace, so settings like TransportType can be
	// shared between all instances in a container .env file. Settings in all other namespaces must be fully specified.
	if (CanSetWithoutNamespaceList.Contains(settingsNamespace))
	{
	var nameOnly = regex.Replace(name, "_");
	yield return nameOnly.ToUpperInvariant();
	yield return nameOnly;
	}
	}
	static readonly HashSet<SettingsRootNamespace> CanSetWithoutNamespaceList =
	[
	new SettingsRootNamespace("ServiceControl"),
	new SettingsRootNamespace("ServiceControl.Audit"),
	new SettingsRootNamespace("Monitoring")
	];

This means we can't use it with a prefix as that logic would still load the value directly from OTEL_EXPORTER_OTLP_ENDPOINT.

[ramonsmits]

I think this needs to be added to AuditMetrics

[ramonsmits]

I think this should better define its intend:

• No need for Audit in the name as the namespace already is clear
• Its for OpenTelemetry
• These are constants

Suggested change

	static class AuditMetrics
	{
	public const string MeterName = "Particular.ServiceControl";
	public static readonly Meter Meter = new(MeterName, "0.1.0");
	public static readonly string Prefix = "particular.servicecontrol.audit";
	}
	static class TelemetryGlobals
	{
	public const string Source = "Particular.ServiceControl";
	public const string Scope = "particular.servicecontrol.audit";
	public static readonly Meter Meter = new(MeterName, "0.1.0");
	}

[ramonsmits]

See other comment on OTEL_EXPORTER_OTLP_ENDPOINT

Suggested change

	public string OtelMetricsUrl { get; set; } = SettingsReader.Read<string>(SettingsRootNamespace, nameof(OtelMetricsUrl));
	const string OTEL_EXPORTER_OTLP_ENDPOINT = nameof(OTEL_EXPORTER_OTLP_ENDPOINT);
	[]
	public string OtelExporterOltpEndpointUrl { get; init; } = SettingsReader.Read<string>(SettingsRootNamespace, OTEL_EXPORTER_OTLP_ENDPOINT);

[ramonsmits]

There are a lot of things done in this method. Even multiple measurements.
I think we should consider doing the measures in the unitOfWork methods itself. The measurement is then simply that whole method and isn't making this method even worse then it already is :-)

[andreasohlund]

@lailabougria ready for review