Skip to content
This repository has been archived by the owner on Oct 3, 2023. It is now read-only.

Propagation fails if a new root span is not sampled. #119

Closed
tcolgate opened this issue Sep 3, 2018 · 5 comments
Closed

Propagation fails if a new root span is not sampled. #119

tcolgate opened this issue Sep 3, 2018 · 5 comments

Comments

@tcolgate
Copy link
Contributor

tcolgate commented Sep 3, 2018

If a new root span is created by startRootSpan, but the decision is made not to sample it, the rootSpan is not passed on (instead null is passed). This results in the propagation code attempting to dereference null to read a spanContext (propagating an unsampled span context is valid behaviour I believe, so that the no-sample decision is passed on).

see:
opencensus-core/src/trace/model/tracer.ts:132
and:
instrumentation-http/build/src/http.js:186

The instrumentation handler should probably be more cautious, but the root span creation should also be passing on the new span.

@tcolgate
Copy link
Contributor Author

tcolgate commented Sep 3, 2018

On further investigation, the gRPC instrumentation avoids the problem doing a NoOp if span is null.
From what I have seen elsewhere, this is not the expected behaviour (unsampled spans should still have context assigned, and be propagated, they just not be exported).

@tcolgate
Copy link
Contributor Author

tcolgate commented Sep 3, 2018

#120

@nickform
Copy link

nickform commented Nov 6, 2018

I am curious if I am hitting this issue. Yesterday, I added this change to a node server which was already successfully exporting traces to stackdriver:

 import * as tracing from '@opencensus/nodejs';
 import * as tracingStackdriverExporter from '@opencensus/exporter-stackdriver';
+import * as tracingStackdriverPropagation from '@opencensus/propagation-stackdriver';

@@ -54,7 +55,8 @@ const nodeCleanup = require('node-cleanup');

 export function configureServer(app: express.Application, server: http.Server) {
   const tracingExporter = new tracingStackdriverExporter.StackdriverTraceExporter({ projectId: process.env.GOOGLE_PROJECT });
-  tracing.start({ exporter: tracingExporter });
+  const tracingPropagation = tracingStackdriverPropagation.v1;
+  tracing.start({ exporter: tracingExporter, propagation: tracingPropagation });

And I immediately started to get Cannot read property 'spanContext' of null messages followed by the Error: socket hang up mentioned in #110 .

After turning the logging up, I saw a bunch of messages like calling RootSpan.startSpan() on ended RootSpan which I tracked down to this line in what seems to be the core tracing package.

Btw, I also tracked down this line as the source of the Cannot read property 'spanContext' of null messages in case that's useful.

Caveat: the links above aren't necessarily links to the exact same commits as the version I'm using. For reference, my package.json declares the following opencensus dependencies:

    "@opencensus/exporter-stackdriver": "0.0.4",
    "@opencensus/instrumentation-grpc": "0.0.5",
    "@opencensus/nodejs": "0.0.4",
    "@opencensus/propagation-stackdriver": "0.0.6",

Can one of the package maintainers give me some guidance on whether my symptoms are similar enough to this issue? If not, I will open a separate issue.

Thanks!

@tcolgate
Copy link
Contributor Author

tcolgate commented Nov 7, 2018

Closing this. The initial bug was fixed, the wider issues of span propagation remain.

@tcolgate tcolgate closed this as completed Nov 7, 2018
@nickform
Copy link

nickform commented Nov 7, 2018

@tcolgate - any advice on how to follow up on the issue I'm having?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants