Support W3C distributed tracing format

[lmolkova]

W3C distributed tracing protocol is going to replace current Request-Id format.

The implementation should be split into several phases

1. Start emitting w3c-compatible operation ids, without supporting W3C headers.
   This phase should be implemented first and phase 2 could follow in several months.

2. Opt-in support for W3C and Request-Id
   Once we are sure p2 works well across different SDKs and don't break customers, we could switch to p3

3. Supporting W3C enabled by default, in back-compat mode with Request-Id

4. Deprecating and retiring Request-Id (in several years)

1. W3C compatible ids

In order to enable customer transition story and backward compatibility, the first step is to start emitting w3c-compatible ids:
32 [a-z][0-9] lowercase characters (i.e. lowercase guid without dashes or 16 random bytes encoded into hex string). All zeros is considered invalid.
Example of the valid id: 0af7651916cd43dd8448eb211c80319c.

The rest: header names and hierarchy in Request-Id could still be preserved.

If request-id was received from upstream, the SDK should continue to behave the same - propagate and do not attempt to change the format.

So, from Java SDKs perspective, the only change here seems to be TelemetryCorrelationUtils.generateRootId function

https://github.com/Microsoft/ApplicationInsights-Java/blob/master/web/src/main/java/com/microsoft/applicationinsights/web/internal/correlation/TelemetryCorrelationUtils.java#L392

2. Supporting W3C (opt-in) and Request-Id.

there are several existing implementations that could be used as reference: .NET AI SDK, Python OpenCensus.

There should be a setting that enables W3C support.

When W3C is enabled:

1. If incoming request contains traceparent only: parse it, set operation, parent and telemetry id accordingly. traceparent should be injected into outgoing requests. request-id should be injected in back-compat format |traceId.spanId. so that older SDKs could understand it.
2. If incoming request contains only Request-Id, new traceId and spanId should be started, Request-Id value should be used as a parent. Root part of Request-Id should be stored in custom dimension on the request telemetry if root part is different from traceId. On the outgoing request side, Request-Id should be emitted in the |traceId.spanId. format
3. if request contains both: request-id and traceparent, request-id should be ignored, so it becomes the same as p1
4. if request does not contain any correlation headers, see case p2

[dhaval24]

@lmolkova thanks for starting the discussion. We should brainstorm this proposal! Also I think we as team should think about how, we are going to design P2 well in advance (when making changes for p1 ideally) so that the architecture is malleable enough.

[lmolkova]

at this point, I'm asking for p1 only. W3C is now being implemented in OpenCensus for Java, we can reuse some of the code or at least ideas from there. And I agree we should design p2 well and not rush with the implementation.

[dhaval24]

@lmolkova makes sense. Lets see if, the change is small enough and could be taken into next release of the SDK but @grlima I think has better understanding on this and I will let him answer this (though I am personally very interested in knowing the deeper happenings here :))

[grlima]

@lmolkova - thanks for creating the issue. We will take this into consideration when we start making progress toward the Java SDK -> OC integration.

[lmolkova]

@grilma, AI SDK for .NET already starts adopting W3C protocol and phase 1 is essential to make a correlation between Java and .NET SDK work. So please prioritize it accordingly.

[dhaval24]

completed with #785