[CONTINT-3554] Send the Datadog-Entity-ID header, containing either the container-id or the cgroup inode if available

[AliDatadog]

What does this PR do?

This PR sends the Datadog-Entity-ID header when the Datadog-Container-Id is usually sent, containing either the container-id or the cgroup inode if available.

Motivation

Using the cgroup inode will allow the trace-agent to retrieve the container-id, typically on cgroupv2 without UDS.

Drawbacks

Datadog-Entity-ID will be sent even if the app is not running in a container because there is no reliable way to ensure that the app is running in a container.

Test

This PR should be tested in several setups without UDS:

• Only cgroupv2 is available: Traces should contain the in-<inode> with the inode obtain by kubectl exec <pod> -- stat /sys/fs/cgroup
• Hybrid setup: If the container-id was available before this PR, it should still be available. Otherwise it should use the cgroup memory controller inode (kubectl exec <pod> -- stat /sys/fs/cgroup/memory).

With UDS, we should never use the inode logic.

• Cgroupv1: Same as hybrid

• To test e2e, use an image containing this PR [CONTINT-3554] Send the Datadog-Entity-ID header, containing either the container-id or the cgroup inode if available #2402 or directly docker.io/alidatadog/agent:7-51-0-trace-containerid on a kind 1.28 cluster using the following manifest:

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: dummy-apm-app
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: dummy-apm-app
    template:
      metadata:
        labels:
          app: dummy-apm-app
          admission.datadoghq.com/enabled: "false"
      spec:
        containers:
        - name: dummy-apm-app
          image: docker.io/alidatadog/dummy-apm-app:0.2.0
          env:
          - name: DD_AGENT_HOST
            valueFrom:
              fieldRef:
                fieldPath: status.hostIP

• Otherwise I made a dummy app that logs span headers. It should be deployed on kind 1.28 as well. The docker vm should use cgroupv2 by default.

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: dummy-apm-app
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: dummy-apm-app
    template:
      metadata:
        labels:
          app: dummy-apm-app
          admission.datadoghq.com/enabled: "false" # Important otherwise container_id won't be shared.
      spec:
        containers:
        - name: dummy-apm-app
          image: docker.io/alidatadog/dummy-apm-app:0.1.0

Example:

2023/11/30 11:10:50 map[Accept-Encoding:[gzip] Content-Type:[application/msgpack] Datadog-Client-Computed-Top-Level:[yes] Datadog-Client-Dropped-P0-Spans:[0] Datadog-Client-Dropped-P0-Traces:[0] Datadog-Entity-Id:[in-35385] Datadog-Meta-Lang:[go] Datadog-Meta-Lang-Interpreter:[gc-arm64-linux] Datadog-Meta-Lang-Version:[1.21.4] Datadog-Meta-Tracer-Version:[v1.59.0-dev] User-Agent:[Go-http-client/1.1] X-Datadog-Trace-Count:[1]]

• If you're interested by the app, I used the following code:

package main

import (
	"log"
	"net/http"
	"time"

	"gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)

func main() {
	go func() {
		http.HandleFunc("/v0.4/traces", func(w http.ResponseWriter, r *http.Request) {
			log.Println(r.Header)
		})
		log.Fatal(http.ListenAndServe(":8080", nil))
	}()

	tracer.Start(
		tracer.WithAgentAddr(":8080"),
		tracer.WithService("my.service"),
	)
	defer tracer.Stop()

	ticker := time.NewTicker(10 * time.Second)
	for ; true; <-ticker.C {
		span := tracer.StartSpan("my.span")
		time.Sleep(2 * time.Second)
		span.Finish()
	}
}

and build the image with the following Dockerfile

# Use the official Golang image to create a build artifact.
# This is based on Debian and sets the GOPATH to /go.
FROM golang:latest as builder

# Set the Current Working Directory inside the container
WORKDIR /app

# Copy the source from the current directory to the Working Directory inside the container
COPY . .

# Build the Go app
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o main .

# Command to run the executable
CMD ["./main"]

To build a multi-arch image, I used docker buildx build --push --platform=linux/amd64,linux/arm64 -t docker.io/alidatadog/dummy-apm-app:0.1.0 .

Reviewer's Checklist

• Changed code has unit tests for its functionality at or near 100% coverage.
• System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
• There is a benchmark for any new code, or changes to existing code.
• If this interacts with the agent in a new way, a system test has been added.
• Add an appropriate team label so this PR gets put in the right place for the release notes.

For Datadog employees:

• If this PR touches code that handles credentials of any kind, such as Datadog API keys, I've requested a review from @DataDog/security-design-and-guidance.
• This PR doesn't touch any of that.

Unsure? Have a question? Request a review!

[pr-commenter]

Benchmarks

Benchmark execution time: 2023-12-14 16:54:03

Comparing candidate commit bd1d7cd in PR branch ali/add-entity-id-header with baseline commit 5a92f7b in branch main.

Found 5 performance improvements and 0 performance regressions! Performance is the same for 34 metrics, 2 unstable metrics.

scenario:BenchmarkPartialFlushing/Disabled-24

• 🟩 execution_time [-9.804ms; -5.840ms] or [-3.395%; -2.022%]

scenario:BenchmarkPartialFlushing/Enabled-24

• 🟩 execution_time [-11.606ms; -7.114ms] or [-3.998%; -2.451%]

scenario:BenchmarkSingleSpanRetention/no-rules-24

• 🟩 execution_time [-10.086µs; -9.256µs] or [-3.964%; -3.638%]

scenario:BenchmarkSingleSpanRetention/with-rules/match-all-24

• 🟩 execution_time [-10.958µs; -9.832µs] or [-4.262%; -3.825%]

scenario:BenchmarkSingleSpanRetention/with-rules/match-half-24

• 🟩 execution_time [-10.451µs; -9.040µs] or [-4.071%; -3.521%]

[ahmed-mez]

Looks great, thanks! Let's get this merged once we have the changes in the agent!

[piochelepiotr]

LGTM for data streams

[nsrip-dd]

Thank you! One small request regarding the profiler unit tests. Otherwise LGTM.

[nsrip-dd]

LGTM for profiling, thanks!

[vboulineau]

I think code is incorrect here, mountsPath is /proc/mounts, which is not read as we defaulted to always reading /sys/fs/cgroup, it should be defaultCgroupMountPath and mountsPath should be removed.

Note that you have no unit test on that.

[AliDatadog]

Nice catch. I fixed it and added a unit test

[ahmed-mez]

@AliDatadog @vboulineau is this ready to merge?

[AliDatadog]

@AliDatadog @vboulineau is this ready to merge?

Waiting for the last approval

[vboulineau]

Not used, should be removed