unexpected metrics for BenchmarkBuildSecret

[crazy-max]

Since https://moby.github.io/buildkit-bench/#/result/20240918-081354, BenchmarkBuildSecret:

buildkit-bench/test/build_test.go

Lines 76 to 93 in 114bb20

	// https://github.com/docker/buildx/issues/2479
	func benchmarkBuildSecret(b *testing.B, sb testutil.Sandbox) {
	dockerfile := []byte(`
	FROM python:latest
	RUN --mount=type=secret,id=SECRET cat /run/secrets/SECRET
	`)
	dir := tmpdir(
	b,
	fstest.CreateFile("Dockerfile", dockerfile, 0600),
	fstest.CreateFile("secret.txt", []byte("mysecret"), 0600),
	)
	b.ResetTimer()
	b.StartTimer()
	out, err := buildxBuildCmd(sb, withDir(dir), withArgs("--secret=id=SECRET,src=secret.txt", "."))
	b.StopTimer()
	sb.WriteLogFile(b, "buildx", []byte(out))
	require.NoError(b, err, out)
	}

does not return what we would expect.

Previously: https://moby.github.io/buildkit-bench/#/result/20240917-213205

Current: https://moby.github.io/buildkit-bench/#/result/20240918-081354

We should see the regression between v0.12.5 and v0.13.0 related to moby/buildkit#4624 as shown in previous benchs:

But we don't since https://moby.github.io/buildkit-bench/#/result/20240918-081354

Changes we made between 20240917-213205 and 20240918-081354 doesn't seem related:

• buildkit-bench/result/20240917-213205/gha-event.json

  Line 36 in be892a7

  "id": "88ac7cbffcd401ca5344d3a14c840c8a66910da5",

• buildkit-bench/result/20240918-081354/gha-event.json

  Line 36 in be892a7

  "id": "1b615403b7f988a1041d5a4a10287848ccd2d215",

88ac7cb...1b61540