Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

opentelemetry-instrumentation-aws-lambda | Invalid type NoneType for attribute 'http.user_agent' value. #1780

Closed
imanshafiei540 opened this issue May 3, 2023 · 4 comments · Fixed by #1785
Labels
bug Something isn't working good first issue Good for newcomers help wanted Extra attention is needed

Comments

@imanshafiei540
Copy link
Contributor

imanshafiei540 commented May 3, 2023

Environment:

I'm using opentelemetry-instrumentation-aws-lambda to instrument the Lambda function behind the API Gateway (V2).

Python==3.9
AWS-OTEL Lmabda Layer == arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-python-amd64-ver-1-17-0:1
fastapi==0.73.0
opentelemetry-sdk==1.17.0

Environment variables:

AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument
OPENTELEMETRY_COLLECTOR_CONFIG_FILE=/var/task/src/otel-config.yaml
OTEL_METRICS_EXPORTER=otlp
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OPENTELEMETRY_EXTENSION_LOG_LEVEL=error
OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=fastapi

OTEL Config yaml:

receivers:
  otlp:
    protocols:
      http:

exporters:
  otlp:
    endpoint: https://<ANOTHER_SERVICE>:4317

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [otlp]

Steps to reproduce

  1. Instrument your Lambda with as stated here: https://aws-otel.github.io/docs/getting-started/lambda/lambda-python
  2. Lambda should be behind the API Gateway
  3. Send a request

What is the expected behavior?
There shouldn't be any Warning logs in the Lambda logs.

What is the actual behavior?
I'm seeing couple of Warning in the lambda logs.

Screenshot 2023-05-03 at 9 30 27 AM

[WARNING]	2023-05-02T23:36:45.395Z	ec784472-c5bb-4f74-867d-e45d39e0fcca	Invalid type NoneType for attribute 'http.user_agent' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types

Additional context
I'm interested to open a PR if possible to add some logic here preventing None values being set as attributes: https://github.com/open-telemetry/opentelemetry-python-contrib/blob/46e4b1da44c534fed8e1002899e9e41e6d668018/instrumentation/opentelemetry-instrumentation-aws-lambda/src/opentelemetry/instrumentation/aws_lambda/__init__.py#LL232C3-L232C3

Also, if there is a way to disable Tracing/Warning Logs it would be helpful, I couldn't find a way to do that.

Another related discussion: open-telemetry/opentelemetry-python#3293

Thanks.

@imanshafiei540 imanshafiei540 added the bug Something isn't working label May 3, 2023
@srikanthccv
Copy link
Member

I'm interested to open a PR if possible to add some logic here preventing None values being set as attributes

Feel free to send a pull request.

Also, if there is a way to disable Tracing/Warning Logs it would be helpful, I couldn't find a way to do that.

See open-telemetry/opentelemetry-python#1059

@srikanthccv srikanthccv added good first issue Good for newcomers help wanted Extra attention is needed labels May 3, 2023
@imanshafiei540
Copy link
Contributor Author

@srikanthccv I need some insights about the best way here.
After further investigation, I realized that my Lambda behind the APIGW is working with no problem. The problem is for Lambda behind the ALB.
The logic itself is checking if the version is equal to 2.0 set V2 attributes, otherwise set V1 attributes:

if lambda_event.get(“version”) ==2.0":
    _set_api_gateway_v2_proxy_attributes(lambda_event, span)
else:
    _set_api_gateway_v1_proxy_attributes(lambda_event, span)

Do you think it’s best that I keep this behavior and only add some checking inside the _set_api_gateway_v1_proxy_attributes

Or should I change the behavior by changing else to elif lambda_event.get(“version”) == “1.0"

In addition, I also can add an else statement and provide some kind of default behavior, but I don’t think I could be able to make it so generic.

The problem (The second way) is the current Lambda functions behind the ALB will not have any attributes, even attributes like X-Forwarded-Proto will be removed.

The first way is more like a stable way, but I need to add a couple of if statements to the logic.

Let me know what you think.

Thanks.

@srikanthccv
Copy link
Member

You mentioned a lot of lambda and AWS-specific detail I do not understand. What I could tell is that warning originates from the package, and it's because of .get(). I could suggest to the point that there should be a None check.

@imanshafiei540
Copy link
Contributor Author

imanshafiei540 commented May 5, 2023

You mentioned a lot of lambda and AWS-specific detail I do not understand. What I could tell is that warning originates from the package, and it's because of .get(). I could suggest to the point that there should be a None check.

Thank you. I made those changes. Now, waiting for EasyCLA. Let me know if I need to do anything in between.

UPDATE:

@srikanthccv The EasyCLA is OK now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
2 participants