dd-trace >= 4.7.0 crashes with TypeError: Cannot read properties of undefined (reading 'span')

[AaronMoat]

Expected behaviour

Expected no change to functionality in minor upgrade

Actual behaviour

My app fails on startup with this error:

TypeError: Cannot read properties of undefined (reading 'span')
--
at Http2ClientPlugin._onResponse (/node_modules/.pnpm/dd-trace@4.7.0/node_modules/dd-trace/packages/datadog-plugin-http2/src/client.js:98:11)
at Http2ClientPlugin.bindAsyncStart (/node_modules/.pnpm/dd-trace@4.7.0/node_modules/dd-trace/packages/datadog-plugin-http2/src/client.js:78:14)
at /node_modules/.pnpm/dd-trace@4.7.0/node_modules/dd-trace/packages/dd-trace/src/plugins/tracing.js:71:59
at StoreBinding._transform (/node_modules/.pnpm/dd-trace@4.7.0/node_modules/dd-trace/packages/dd-trace/src/plugins/plugin.js:35:11)
at /node_modules/.pnpm/dd-trace@4.7.0/node_modules/dd-trace/packages/diagnostics_channel/src/index.js:98:17
at Channel.runStores.ActiveChannelPrototype.runStores (/node_modules/.pnpm/dd-trace@4.7.0/node_modules/dd-trace/packages/diagnostics_channel/src/index.js:86:12)
at ClientHttp2Stream.emit (/node_modules/.pnpm/dd-trace@4.7.0/node_modules/dd-trace/packages/datadog-instrumentations/src/http2/client.js:19:32)
at emit (node:internal/http2/core:330:3)
at process.processTicksAndRejections (node:internal/process/task_queues:85:22)

Steps to reproduce

Unsure on exact reproduction steps as some internal services appear to be unaffected. In particular the same service with additional opentelemetry (from some surrounding containers) disabled functions fine.

Environment

• Operation system: Amazon Linux (ECS Fargate), running Docker node:18.16.0-slim
• Node.js version: v18.16.0
• Tracer version: 4.7.0 (upgraded from 4.6.0) onwards
• Agent version: datadog/agent:7
• Relevant library versions: n/a

[AaronMoat]

@rochdev @tlhunter apologies for the direct ping, but I think this relates to #3284 - the stack trace is all in that changed area of this PR for 4.7.0. Any chance you could take a look or advise if I can provide more specific information to help diagnose?

[rochdev]

@AaronMoat Looking at the code, the only way I can see for this to happen is that a request is started before the instrumentation is applied. This should not be possible if the tracer is initialized before anything else happens in the process, but it can happen otherwise.

I'll work on a fix since even when that's the case the tracer shouldn't throw, but then I would still expect traces to not be reported properly even after the fix has been released since the tracer relies on being initialized first. Can you make sure that dd-trace initialization is the very first thing that happens in your app, before any other code or any other imports? An easy way to make sure of this is to initialize directly from the CLI with NODE_OPTIONS='-r dd-trace/init' ./server.js.

[AaronMoat]

Thanks @rochdev -- I believe that's all set up correctly, we can't import dd-trace/init due to providing some configuration not configurable via logger, however the code to setup is the first import.

[AaronMoat]

Hi @rochdev, I'm still getting this crash in 4.11.

TypeError: Cannot read properties of undefined (reading 'span')
at Http2ClientPlugin._onResponse (/node_modules/.pnpm/dd-trace@4.11.0/node_modules/dd-trace/packages/datadog-plugin-http2/src/client.js:98:11)

[clintonb]

I'm also seeing this issue with 4.11.0. I upgraded from 3.20.0, and never saw this error with that version.

Related: #2915

We see this issue for 4.11.x with both Node 16 and 20.