Update StructFieldConst to find missing offsets at runtime

[MrAlias]

If the offset index does not contain an entry for the desired struct field, use the target process DWARF data to determine the offset.

Testing

End-to-end testing was done by updating the autosdk e2e test. All dependencies were bumped to commit hash versions:

0c18ff3#diff-14bffadf6f47d1ef427c47d4425fca90398e7b4c8d96f0e58f1a371833d837ebR6-R8

It was then tested both locally and on our CI system.

Locally the logs were verified to show this was working:

{"time":"2025-01-13T20:42:48.422927384Z","level":"INFO","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":490},"msg":"Offset not cached, analyzing directly","key":"span_context_trace_id_pos","id":{"ModPath":"go.opentelemetry.io/otel","PkgPath":"go.opentelemetry.io/otel/trace","Struct":"SpanContext","Field":"traceID"}}
{"time":"2025-01-13T20:42:48.460714317Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":508},"msg":"Offset found","key":"span_context_trace_id_pos","id":{"ModPath":"go.opentelemetry.io/otel","PkgPath":"go.opentelemetry.io/otel/trace","Struct":"SpanContext","Field":"traceID"},"offset":0}
{"time":"2025-01-13T20:42:48.46075608Z","level":"INFO","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":490},"msg":"Offset not cached, analyzing directly","key":"span_context_span_id_pos","id":{"ModPath":"go.opentelemetry.io/otel","PkgPath":"go.opentelemetry.io/otel/trace","Struct":"SpanContext","Field":"spanID"}}
{"time":"2025-01-13T20:42:48.499132571Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":508},"msg":"Offset found","key":"span_context_span_id_pos","id":{"ModPath":"go.opentelemetry.io/otel","PkgPath":"go.opentelemetry.io/otel/trace","Struct":"SpanContext","Field":"spanID"},"offset":16}
{"time":"2025-01-13T20:42:48.499175132Z","level":"INFO","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":490},"msg":"Offset not cached, analyzing directly","key":"span_context_trace_flags_pos","id":{"ModPath":"go.opentelemetry.io/otel","PkgPath":"go.opentelemetry.io/otel/trace","Struct":"SpanContext","Field":"traceFlags"}}
{"time":"2025-01-13T20:42:48.539235866Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":508},"msg":"Offset found","key":"span_context_trace_flags_pos","id":{"ModPath":"go.opentelemetry.io/otel","PkgPath":"go.opentelemetry.io/otel/trace","Struct":"SpanContext","Field":"traceFlags"},"offset":24}

The update was then reverted given it will be managed by renovate.

[RonFed]

This is awesome

[MrAlias]

DWARF data caching analysis

I wrote a benchmark to evaluate the overhead of reading the DWARF data:

package dwarfsize

import (
	"debug/dwarf"
	"debug/elf"
	"reflect"
	"testing"
	"unsafe"
)

func Benchmark(b *testing.B) {
	b.Run("otelcontribcol", benchmark("./bin/otelcontribcol_linux_amd64"))
	b.Run("autosdk", benchmark("./bin/autosdk"))
}

func benchmark(path string) func(*testing.B) {
	return func(b *testing.B) {
		f, err := elf.Open(path)
		if err != nil {
			b.Fatalf("failed to open ELF: %s", path)
		}
		defer f.Close()

		data, err := f.DWARF()
		if err != nil {
			b.Fatalf("failed to read DWARF symbols: %s", err)
		}

		if data == nil {
			b.Fatalf("empty DWARF data")
		}

		size := dwarfSize(*data)
		b.Logf("DWARF data size: %d bytes", size)

		b.ReportAllocs()
		b.ResetTimer()

		for n := 0; n < b.N; n++ {
			data, err = f.DWARF()
			if err != nil {
				b.Fatal(err)
			}
		}

		_ = data
	}
}

func dwarfSize(data dwarf.Data) uintptr {
	size := unsafe.Sizeof(data)

	v := reflect.ValueOf(data)
	for i := 0; i < v.NumField(); i++ {
		field := v.Field(i)
		switch field.Type().Kind() {
		case reflect.Array, reflect.Chan, reflect.Slice:
			size += uintptr(field.Cap()) * field.Type().Size()
		case reflect.Map, reflect.String:
			size += uintptr(field.Len()) * field.Type().Size()
		default:
		}
	}

	return size
}

I evaluated this against the autosdk e2e test binary and the OpenTelemetry collector contrib binary:

$ ls -lh bin
-rwxr-xr-x 1 tyler tyler 3.0M Jan 14 11:38 autosdk
-rwxr-xr-x 1 tyler tyler 411M Jan 14 11:15 otelcontribcol_linux_amd64

Results

goos: linux
goarch: amd64
pkg: testing/dwarfsize
cpu: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz
                  │   out.txt   │
                  │   sec/op    │
/otelcontribcol-8    2.241 ± 5%
/autosdk-8          21.55m ± 2%
geomean             219.8m

                  │   out.txt    │
                  │     B/op     │
/otelcontribcol-8   1.353Gi ± 0%
/autosdk-8          3.420Mi ± 0%
geomean             68.84Mi

                  │   out.txt   │
                  │  allocs/op  │
/otelcontribcol-8   315.7k ± 0%
/autosdk-8          3.115k ± 0%
geomean             31.36k

The logged dwarf data size:

• otelcontribcol: DWARF data size: 5093228696 bytes
• autosdk: DWARF data size: 39317984 bytes

Take-away

If my analysis is correct, it looks like this can allocate up to a few GB for binaries with large DWARF data and also take a few seconds to process that data.

I plan to remove the caching based on this analysis. It seems more reasonable to have the auto-instrumentation take a longer time to start in the failure-case that we do not know an offset or two than it is to have the resident memory of the auto-instrumentation be several GB for its lifetime.

We can always readdress this in the future if needed.

[RonFed]

@MrAlias Thanks for doing this benchmark, it's really interesting.
I agree that having the DWARF cached is not a good idea.
I wonder if we can do anything to reduce the memory allocation / latency for that DWARF processing.