question: Distributed tracing with contrib packages gorm and fiber

[dominicfollett]

Hello there, I'm looking for a bit of guidance, with a caveat: that I am still learning about tracing and my use of these contrib packages is somewhat fresh to me, so I apologize in advance if any of my questions are a bit off the mark.

I have three services that are successfully reporting traces, but the traces are not correlated when they should be, and they are not 'connected' on the Datadog service map.

Services (Service C calls into A, which does a look up on B.):

• A: instrumented fiber api
• B: instrumented gorm db
• C: instrumented js client

I am wondering whether I need to do more than the basic setup of the contrib tracing packages. I am hoping to see correlation between C and A, and A and B.

This currently what I have:

Current fiber setup:

func main() {
  tracer.Start()
  defer tracer.Stop()
  ...
  // Add fibertrace Middleware
  api.Use(fibertrace.Middleware(
    fibertrace.WithServiceName("service-A"),
  ))

Gorm setup

gormtrace.Open(postgres.New(postgres.Config{DSN: dsn}), &gorm.Config{})

Additionally I have set the propagation environment variables:

DD_PROPAGATION_STYLE_INJECT=Datadog,B3
DD_PROPAGATION_STYLE_EXTRACT=Datadog,B3

And I can confirm that service A is receiving the parent trace and span id's from service C on the headers:

X-Datadog-Trace-Id: 5663109102938475867 // example ids
X-Datadog-Parent-Id: 3962811275849372967

Is it possible that I need to explicitly set the propagation style as a fibertrace middleware option or is having it set as an environment variable sufficient? And do I need to somehow pass fibertrace's span id onto the gorm tracer? And if so, how would I be able to do that? I have looked through the docs but I haven't found any examples that might help me.

Again, apologies if these question are unclear. And thank you for your help.

[ajgajg1134]

Hello 👋! So if I understand correctly your service map looks like C calls A which Calls B? (C -> A -> B)
And you see the proper propagation from C to A but not to B?

In your usage of gorm are you providing the context when you make a query? e.g. .WithContext(ctx) where the context is the one coming from service A (which carries the tracing information)?

Sorry this answer has been a bit slow getting to you. Feel free to also open a support ticket directly with datadog if the above doesn't help (and will ensure you get a more timely response)

[dominicfollett]

Hi there @ajgajg1134 thanks for the response.

The service map looks like (C -> A ->B) correct, but I am not seeing proper propagation from C to A, nor from A to B.
Incidentally I did try .WithContext(ctx) on the DB queries, but was not observing any correlation either.

All I can confirm is that the required DD trace headers are been correctly sent from C to A.

I'm at a loss as to what else to try. It would be really awesome if we could get correlated traces for these services.

[ajgajg1134]

Hi @dominicfollett, this is tough to diagnose without seeing more code or looking at the spans themselves. I recommend you open a support ticket directly with datadog so a support agent can take a look at the spans themselves to see if it's a bug in our code somewhere or a subtle configuration issue

[dominicfollett]

Thank you @ajgajg1134 for your help all the same, I will do that.