opentelemetry-instrumentation-aws-lambda | Invalid type NoneType for attribute 'http.user_agent' value.

[imanshafiei540]

Environment:

I'm using opentelemetry-instrumentation-aws-lambda to instrument the Lambda function behind the API Gateway (V2).

Python==3.9
AWS-OTEL Lmabda Layer == arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-python-amd64-ver-1-17-0:1
fastapi==0.73.0
opentelemetry-sdk==1.17.0

Environment variables:

AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument
OPENTELEMETRY_COLLECTOR_CONFIG_FILE=/var/task/src/otel-config.yaml
OTEL_METRICS_EXPORTER=otlp
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OPENTELEMETRY_EXTENSION_LOG_LEVEL=error
OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=fastapi

OTEL Config yaml:

receivers:
  otlp:
    protocols:
      http:

exporters:
  otlp:
    endpoint: https://<ANOTHER_SERVICE>:4317

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [otlp]

Steps to reproduce

1. Instrument your Lambda with as stated here: https://aws-otel.github.io/docs/getting-started/lambda/lambda-python
2. Lambda should be behind the API Gateway
3. Send a request

What is the expected behavior?
There shouldn't be any Warning logs in the Lambda logs.

What is the actual behavior?
I'm seeing couple of Warning in the lambda logs.

[WARNING]	2023-05-02T23:36:45.395Z	ec784472-c5bb-4f74-867d-e45d39e0fcca	Invalid type NoneType for attribute 'http.user_agent' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types

Additional context
I'm interested to open a PR if possible to add some logic here preventing None values being set as attributes: https://github.com/open-telemetry/opentelemetry-python-contrib/blob/46e4b1da44c534fed8e1002899e9e41e6d668018/instrumentation/opentelemetry-instrumentation-aws-lambda/src/opentelemetry/instrumentation/aws_lambda/__init__.py#LL232C3-L232C3

Also, if there is a way to disable Tracing/Warning Logs it would be helpful, I couldn't find a way to do that.

Another related discussion: open-telemetry/opentelemetry-python#3293

Thanks.

[srikanthccv]

I'm interested to open a PR if possible to add some logic here preventing None values being set as attributes

Feel free to send a pull request.

Also, if there is a way to disable Tracing/Warning Logs it would be helpful, I couldn't find a way to do that.

See open-telemetry/opentelemetry-python#1059

[imanshafiei540]

@srikanthccv I need some insights about the best way here.
After further investigation, I realized that my Lambda behind the APIGW is working with no problem. The problem is for Lambda behind the ALB.
The logic itself is checking if the version is equal to 2.0 set V2 attributes, otherwise set V1 attributes:

if lambda_event.get(“version”) == “2.0":
    _set_api_gateway_v2_proxy_attributes(lambda_event, span)
else:
    _set_api_gateway_v1_proxy_attributes(lambda_event, span)

Do you think it’s best that I keep this behavior and only add some checking inside the _set_api_gateway_v1_proxy_attributes

Or should I change the behavior by changing else to elif lambda_event.get(“version”) == “1.0"

In addition, I also can add an else statement and provide some kind of default behavior, but I don’t think I could be able to make it so generic.

The problem (The second way) is the current Lambda functions behind the ALB will not have any attributes, even attributes like X-Forwarded-Proto will be removed.

The first way is more like a stable way, but I need to add a couple of if statements to the logic.

Let me know what you think.

Thanks.

[srikanthccv]

You mentioned a lot of lambda and AWS-specific detail I do not understand. What I could tell is that warning originates from the package, and it's because of .get(). I could suggest to the point that there should be a None check.

[imanshafiei540]

You mentioned a lot of lambda and AWS-specific detail I do not understand. What I could tell is that warning originates from the package, and it's because of .get(). I could suggest to the point that there should be a None check.

Thank you. I made those changes. Now, waiting for EasyCLA. Let me know if I need to do anything in between.

UPDATE:

@srikanthccv The EasyCLA is OK now.