0.35.0 trace autologging provides incorrect trace IDs

[hedss]

What's wrong?

Prior to v0.35.0, autologging correctly output loglines with the trace ID that was passed to it, eg:

  | 2023-07-21 11:21:24.493 | span=requester dur=12181760ns status=STATUS_CODE_OK http.method=GET http.target=unicorn svc=mythical-requester traceId=be8481dd93f1c751bd1ee320e5629197
-- | -- | --

From 0.35.0, the trace ID is now converted into what appears to be a decimal only identifer (that is not convertable to a working HEX trace ID hash):

2023-07-21 11:25:47.392 | span=requester dur=27946240ns status=Ok http.method=GET http.target=beholder svc=mythical-requester traceId=3661626132633865396432653963323538376566646565303064373233613266
-- | --

This means that correlation between traces that generate autologged log lines is not longer possible and breaks the general log->traces workflow.

Steps to reproduce

Install Agent v0.35.0 into any ecosystem and then configure it to receive traces and autogenerate logs from traces.

You can easily test this out using the Intro to MLT sandbox, and changing this line to image: grafana/agent:v0.35.0, then use the docker-compose.yaml file to initialise a working system.

Go to the Explorer to see the result and inability to correlate with traces.

System information

MacOS 12.6.5 (21G531)

Software version

Grafana Agent v0.35.0

Configuration

automatic_logging:
        # Use the logs instance defined at the start of the configuration file, specifying a logs instance and the
        # named Promtail instance.
        backend: logs_instance
        logs_instance_name: loki
        # Ensure one log one line per root span (ie. one per trace).
        roots: true
        processes: false
        # Setting spans to `true` (default false) will generate a log line *per span* received. Note that this
        # can produce a huge amount of log information.
        spans: false
        # If present in the span, add the http.method, http.target and http.status_code span attributes to the log line.
        span_attributes:
          - http.method
          - http.target
          - http.status_code
        # Force the trace ID to be set as `traceId`. This ensures correlation from within Grafana between data sources.
        overrides:
          trace_id_key: "traceId"

Logs

2023-07-21 11:25:47.392 | span=requester dur=27946240ns status=Ok http.method=GET http.target=beholder svc=mythical-requester traceId=3661626132633865396432653963323538376566646565303064373233613266
-- | --

[rfratto]

This is likely related to #3858 which updated the OpenTelemetry dependency. cc @ptodev