OpenTelemetry context propagation

[davidkeaveny]

Does anyone know what the current state of play is regarding OpenTelemetry context propagation with AWS.Messaging? i.e. if I call an ASP.NET API, which then pushes a message to the queue, and a subscribing service receives the message and runs its own code, will the original OpenTelemetry context (especially trace ID) be incorporated into the OpenTelemetry context of the subscriber, so I get full end-to-end traceability?

[philasmar]

Could you please take a look at this package https://github.com/awslabs/aws-dotnet-messaging/tree/main/src/AWS.Messaging.Telemetry.OpenTelemetry? You should be able to do what you are asking for using this package. Also, we have some sample apps in the repo that are setting up OTeL.

[davidkeaveny]

Is there any overlap in terms of functionality between AWS.Messaging.Telemetry.OpenTelemetry and OpenTelemetry.Instrumentation.AWSLambda? I'm assuming that the former handles instrumentation of the messaging within the lambda function, while the latter covers the lambda in general. The latter requires a public method that wraps the method being traced, but that seems to clash a bit with how AWS.Messaging.Telemetry.OpenTelemetry wants the ILambdaMessaging dependency injected in the method. This is what I have now:

public class Function
{
    private static readonly ILambdaEntryPoint LambdaEntryPoint = new Startup().ConfigureServices().GetRequiredService<ILambdaEntryPoint>();
    private static readonly TracerProvider Tracer = Sdk.CreateTracerProviderBuilder()
        .AddAWSInstrumentation()
        .AddAWSLambdaConfigurations(options => options.DisableAwsXRayContextExtraction = true)
        .AddOtlpExporter()
        .Build();
    
    private static async Task Main()
    {
        var handler = OpenTelemetryFunctionHandler;
        await LambdaBootstrapBuilder.Create(handler, new SourceGeneratorLambdaJsonSerializer<SqsSerializerContext>())
            .Build()
            .RunAsync();
    }
    
    private static async Task<SQSBatchResponse> OpenTelemetryFunctionHandler(SQSEvent sqsEvent, ILambdaContext lambdaContext)
    {
        return await AWSLambdaWrapper.TraceAsync(Tracer, FunctionHandler, sqsEvent, lambdaContext);
    }

    private static async Task<SQSBatchResponse> FunctionHandler(SQSEvent sqsEvent, ILambdaContext lambdaContext)
    {
        return await LambdaEntryPoint.Handle(sqsEvent, lambdaContext);
    }
}

If I were to add AWS.Messaging.Telemetry.OpenTelemetry, then FunctionHandler would need to take the ILambdaMessging dependency, which would then result in a compile-time error as the AWSLambdaWrapper.TraceAsync can't handle the extra parameter.

Would switching to Amazon.Lambda.Annotations help at all? The examples for AWS.Messaging.Telemetry.OpenTelemetry are using it.

[normj]

I'm assuming new Startup().ConfigureServices() returns back an IServiceProvider. I would suggest saving that into a static field and then in your FunctionHandler have the GetRequiredService<ILambdaMessaging>() call to get the lambda messaging and then forward the Lambda function into ProcessLambdaEventWithBatchResponseAsync. Something like this but I'm ignoring the ILambdaEntryPoint because I'm not exactly sure what that is used for.

public class Function
{
    private static readonly IServiceProvider ServiceProvider = Startup().ConfigureServices();
    private static readonly ILambdaEntryPoint LambdaEntryPoint = new ServiceProvider.GetRequiredService<ILambdaEntryPoint>();
    private static readonly TracerProvider Tracer = Sdk.CreateTracerProviderBuilder()
        .AddAWSInstrumentation()
        .AddAWSLambdaConfigurations(options => options.DisableAwsXRayContextExtraction = true)
        .AddOtlpExporter()
        .Build();
    
    private static async Task Main()
    {
        var handler = OpenTelemetryFunctionHandler;
        await LambdaBootstrapBuilder.Create(handler, new SourceGeneratorLambdaJsonSerializer<SqsSerializerContext>())
            .Build()
            .RunAsync();
    }
    
    private static async Task<SQSBatchResponse> OpenTelemetryFunctionHandler(SQSEvent sqsEvent, ILambdaContext lambdaContext)
    {
        return await AWSLambdaWrapper.TraceAsync(Tracer, FunctionHandler, sqsEvent, lambdaContext);
    }

    private static async Task<SQSBatchResponse> FunctionHandler(SQSEvent sqsEvent, ILambdaContext lambdaContext)
    {
        var messaging = ServiceProvider.GetRequiredService<ILambdaMessaging>();
        return await messaging.ProcessLambdaEventWithBatchResponseAsync(sqsEvent, lambdaContext);
    }
}

[davidkeaveny]

@normj ILambdaEntryPoint is basically a non-Annotations way of adding dependency injection - see https://coderjony.com/blogs/implementing-configuration-logging-dependency-injection-in-aws-lambda-using-net-core for the approach taken.

Thanks for the suggestion above, I will give that a go.