Attempt to fix flakiness in Azure Functions multi-sample build #6521

andrewlock · 2025-01-09T10:46:04Z

Summary of changes

Attempt to fix flakiness in Azure Functions multi-sample build

Reason for change

#6472 added multi-version building for Azure Functions samples. Unfortunately, this caused the build stage to be flaky, because Azure Functions apparently generates and builds a completely separate .csproj. Unfortunately, it also ignores all the artifact output stuff, which means when we restore/build multiple versions of the app (with different api versions, it stomps over itself.

e.g. we can see the sample app build itself doing the right thing:

08:59:05 [DBG]   Samples.AzureFunctions.V4Isolated -> /project/artifacts/bin/Samples.AzureFunctions.V4Isolated/release_net6.0_1.6.0/Samples.AzureFunctions.V4Isolated.dll
08:59:06 [DBG]   Samples.AzureFunctions.V4Isolated -> /project/artifacts/publish/Samples.AzureFunctions.V4Isolated/release_net6.0_1.6.0/

But there's also the "WorkerExtensions" thing which is going completely off the rails, ignores all of the Directory.Build.props etc and just does its own thing

08:59:07 [DBG]   WorkerExtensions -> /tmp/oh4iajvh.kcj/publishout/
08:59:07 [DBG]   WorkerExtensions -> /tmp/zv30zzl2.cxz/publishout/Microsoft.Azure.Functions.Worker.Extensions.dll
08:59:08 [DBG]   WorkerExtensions -> /tmp/zv30zzl2.cxz/publishout/

Unfortunately, this shows it's trying to build two separate versions to the same location, and things break.

Implementation details

Went spelunking through the Azure Functions package code and found Microsoft.Azure.Functions.Worker.Sdk.targets which is doing the actual build. Tried tweaking some of the msbuild variables to convince it to use different locations:

<!-- Setting up some paths we will use for later targets. Broken out into its own as we want to pull it in not just for build (ie: Clean also). -->
  <Target Name="_FunctionsGetPaths">
    <PropertyGroup>
      <_FunctionsMetadataPath>$(IntermediateOutputPath)functions.metadata</_FunctionsMetadataPath>
      <_FunctionsWorkerConfigPath>$(IntermediateOutputPath)worker.config.json</_FunctionsWorkerConfigPath>
      <ExtensionsCsProjDirectory Condition="'$(ExtensionsCsProjDirectory)' == ''">$(IntermediateOutputPath)WorkerExtensions</ExtensionsCsProjDirectory>
      <ExtensionsCsProjDirectory>$([System.IO.Path]::GetFullPath($(ExtensionsCsProjDirectory)))</ExtensionsCsProjDirectory> <!-- Ensure ExtensionsCsProjDirectory is a full path. -->
      <ExtensionsCsProj>$([System.IO.Path]::Combine($(ExtensionsCsProjDirectory), WorkerExtensions.csproj))</ExtensionsCsProj>
      <_FunctionsIntermediateExtensionJsonPath>$(ExtensionsCsProjDirectory)\buildout\bin\$(_FunctionsExtensionsJsonName)</_FunctionsIntermediateExtensionJsonPath>
      <_FunctionsIntermediateExtensionUpdatedJsonPath>$(IntermediateOutputPath)$(_FunctionsExtensionsJsonName)</_FunctionsIntermediateExtensionUpdatedJsonPath>
    </PropertyGroup>
  </Target>

Test coverage

Checked the build logs to make sure it's doing what we expect now

datadog-ddstaging · 2025-01-09T11:31:19Z

Datadog Report

Branch report: andrew/try-fix-azure-functions-build
Commit report: a1c5f0b
Test service: dd-trace-dotnet

✅ 0 Failed, 238073 Passed, 2087 Skipped, 18h 37m 42.78s Total Time
⌛ 1 Performance Regression

⌛ Performance Regressions vs Default Branch (1)

Baseline - Samples.HttpMessageHandler.windows.net80.json.scenarios 268.69ms (+3.47ms, +1%) - Details

andrewlock · 2025-01-09T13:46:34Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6521) - mean (69ms)  : 65, 73
     .   : milestone, 69,
    master - mean (68ms)  : 66, 71
     .   : milestone, 68,

    section CallTarget+Inlining+NGEN
    This PR (6521) - mean (983ms)  : 961, 1005
     .   : milestone, 983,
    master - mean (979ms)  : 957, 1000
     .   : milestone, 979,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6521) - mean (108ms)  : 106, 111
     .   : milestone, 108,
    master - mean (108ms)  : 105, 110
     .   : milestone, 108,

    section CallTarget+Inlining+NGEN
    This PR (6521) - mean (681ms)  : 668, 694
     .   : milestone, 681,
    master - mean (682ms)  : 668, 696
     .   : milestone, 682,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6521) - mean (91ms)  : 89, 94
     .   : milestone, 91,
    master - mean (91ms)  : 89, 93
     .   : milestone, 91,

    section CallTarget+Inlining+NGEN
    This PR (6521) - mean (639ms)  : 624, 655
     .   : milestone, 639,
    master - mean (639ms)  : 625, 654
     .   : milestone, 639,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6521) - mean (195ms)  : 191, 200
     .   : milestone, 195,
    master - mean (195ms)  : 190, 200
     .   : milestone, 195,

    section CallTarget+Inlining+NGEN
    This PR (6521) - mean (1,111ms)  : 1082, 1139
     .   : milestone, 1111,
    master - mean (1,099ms)  : 1062, 1136
     .   : milestone, 1099,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6521) - mean (280ms)  : 276, 284
     .   : milestone, 280,
    master - mean (278ms)  : 274, 283
     .   : milestone, 278,

    section CallTarget+Inlining+NGEN
    This PR (6521) - mean (881ms)  : 856, 906
     .   : milestone, 881,
    master - mean (869ms)  : 838, 899
     .   : milestone, 869,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6521) - mean (270ms)  : 265, 274
     .   : milestone, 270,
    master - mean (266ms)  : 262, 270
     .   : milestone, 266,

    section CallTarget+Inlining+NGEN
    This PR (6521) - mean (859ms)  : 827, 891
     .   : milestone, 859,
    master - mean (845ms)  : 814, 877
     .   : milestone, 845,

andrewlock · 2025-01-10T01:49:19Z

Throughput/Crank Report ⚡

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6521) (11.041M)   : 0, 11041098
    master (11.113M)   : 0, 11113033
    benchmarks/2.9.0 (11.045M)   : 0, 11045405

    section Automatic
    This PR (6521) (7.191M)   : 0, 7190999
    master (7.290M)   : 0, 7290258
    benchmarks/2.9.0 (7.885M)   : 0, 7885346

    section Trace stats
    master (7.533M)   : 0, 7533282

    section Manual
    master (11.036M)   : 0, 11036470

    section Manual + Automatic
    This PR (6521) (6.797M)   : 0, 6797188
    master (6.724M)   : 0, 6723977

    section DD_TRACE_ENABLED=0
    master (10.209M)   : 0, 10208517

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6521) (9.654M)   : 0, 9654283
    master (9.655M)   : 0, 9654974
    benchmarks/2.9.0 (9.586M)   : 0, 9586476

    section Automatic
    This PR (6521) (6.413M)   : 0, 6412798
    master (6.449M)   : 0, 6448807

    section Trace stats
    master (6.838M)   : 0, 6838132

    section Manual
    master (9.552M)   : 0, 9551890

    section Manual + Automatic
    This PR (6521) (6.055M)   : 0, 6055115
    master (5.996M)   : 0, 5996335

    section DD_TRACE_ENABLED=0
    master (9.049M)   : 0, 9049474

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6521) (9.924M)   : 0, 9924160

    section Automatic
    This PR (6521) (6.595M)   : 0, 6595369

    section Manual + Automatic
    This PR (6521) (6.259M)   : 0, 6259423

andrewlock · 2025-01-10T03:28:57Z

Benchmarks Report for tracer 🐌

Benchmarks for #6521 compared to master:

1 benchmarks are faster, with geometric mean 1.141
2 benchmarks are slower, with geometric mean 1.157
1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.04μs	45.4ns	318ns	0.0158	0.0079	0	5.6 KB
master	`StartStopWithChild`	netcoreapp3.1	10.1μs	54.7ns	314ns	0.0193	0.00964	0	5.8 KB
master	`StartStopWithChild`	net472	16.4μs	58.2ns	225ns	1.04	0.303	0.0957	6.21 KB
#6521	`StartStopWithChild`	net6.0	8.06μs	37ns	143ns	0.0158	0.0079	0	5.61 KB
#6521	`StartStopWithChild`	netcoreapp3.1	10.1μs	47.2ns	183ns	0.0146	0.00486	0	5.81 KB
#6521	`StartStopWithChild`	net472	16.6μs	62.8ns	243ns	1.05	0.33	0.0967	6.2 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	497μs	267ns	1.04μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	655μs	404ns	1.51μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	858μs	944ns	3.66μs	0.425	3.3 KB
#6521	`WriteAndFlushEnrichedTraces`	net6.0	468μs	383ns	1.48μs	0	2.7 KB
#6521	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	663μs	440ns	1.59μs	0	2.7 KB
#6521	`WriteAndFlushEnrichedTraces`	net472	853μs	648ns	2.51μs	0.431	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	158μs	1.06μs	10.4μs	0.145	14.47 KB
master	`SendRequest`	netcoreapp3.1	173μs	1.09μs	10.7μs	0.19	17.27 KB
master	`SendRequest`	net472	0.00296ns	0.00105ns	0.00395ns	0	0 b
#6521	`SendRequest`	net6.0	145μs	683ns	3.13μs	0.144	14.47 KB
#6521	`SendRequest`	netcoreapp3.1	173μs	951ns	7.9μs	0.184	17.27 KB
#6521	`SendRequest`	net472	0.00387ns	0.00163ns	0.00632ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #6521

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1	41.29 KB	41.92 KB	633 B	1.53%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	570μs	2.81μs	12.9μs	0.539	0	0	41.76 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	682μs	3.89μs	27.5μs	0.319	0	0	41.29 KB
master	`WriteAndFlushEnrichedTraces`	net472	813μs	3.11μs	12μs	8.41	2.4	0.401	53.31 KB
#6521	`WriteAndFlushEnrichedTraces`	net6.0	590μs	3.23μs	25.8μs	0.551	0	0	41.78 KB
#6521	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	703μs	3.98μs	29.8μs	0.331	0	0	41.92 KB
#6521	`WriteAndFlushEnrichedTraces`	net472	836μs	4.07μs	18.2μs	8.5	2.55	0.425	53.27 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.37μs	1.23ns	4.75ns	0.0143	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.75μs	1.37ns	5.12ns	0.014	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.14μs	4.09ns	15.9ns	0.156	0.00106	987 B
#6521	`ExecuteNonQuery`	net6.0	1.43μs	1.33ns	5.13ns	0.0144	0	1.02 KB
#6521	`ExecuteNonQuery`	netcoreapp3.1	1.79μs	1.69ns	6.34ns	0.0135	0	1.02 KB
#6521	`ExecuteNonQuery`	net472	2.15μs	2.7ns	10.5ns	0.157	0.00107	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.22μs	0.649ns	2.43ns	0.0135	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.54μs	0.413ns	1.49ns	0.013	976 B
master	`CallElasticsearch`	net472	2.65μs	2.16ns	8.08ns	0.158	995 B
master	`CallElasticsearchAsync`	net6.0	1.36μs	0.875ns	3.39ns	0.0129	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.65μs	1.09ns	4.09ns	0.0132	1.02 KB
master	`CallElasticsearchAsync`	net472	2.59μs	1.53ns	5.72ns	0.166	1.05 KB
#6521	`CallElasticsearch`	net6.0	1.28μs	1.23ns	4.75ns	0.0135	976 B
#6521	`CallElasticsearch`	netcoreapp3.1	1.56μs	0.793ns	3.07ns	0.0133	976 B
#6521	`CallElasticsearch`	net472	2.59μs	1.71ns	6.63ns	0.157	995 B
#6521	`CallElasticsearchAsync`	net6.0	1.41μs	0.49ns	1.9ns	0.0135	952 B
#6521	`CallElasticsearchAsync`	netcoreapp3.1	1.69μs	0.568ns	2.13ns	0.0138	1.02 KB
#6521	`CallElasticsearchAsync`	net472	2.71μs	1.76ns	6.83ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.29μs	0.716ns	2.68ns	0.0134	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.62μs	0.453ns	1.63ns	0.0129	952 B
master	`ExecuteAsync`	net472	1.78μs	0.388ns	1.45ns	0.145	915 B
#6521	`ExecuteAsync`	net6.0	1.36μs	0.455ns	1.64ns	0.0131	952 B
#6521	`ExecuteAsync`	netcoreapp3.1	1.66μs	0.958ns	3.58ns	0.0124	952 B
#6521	`ExecuteAsync`	net472	1.82μs	0.507ns	1.9ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.49μs	2.74ns	10.6ns	0.0315	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.31μs	2.41ns	9.02ns	0.0386	2.85 KB
master	`SendAsync`	net472	7.32μs	2.26ns	8.75ns	0.495	3.12 KB
#6521	`SendAsync`	net6.0	4.38μs	1.03ns	3.57ns	0.0308	2.31 KB
#6521	`SendAsync`	netcoreapp3.1	5.17μs	4.04ns	15.7ns	0.0388	2.85 KB
#6521	`SendAsync`	net472	7.42μs	1.93ns	7.46ns	0.494	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.58μs	0.604ns	2.18ns	0.023	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.28μs	0.753ns	2.82ns	0.0216	1.64 KB
master	`EnrichedLog`	net472	2.71μs	1.07ns	4.16ns	0.25	1.57 KB
#6521	`EnrichedLog`	net6.0	1.54μs	1.17ns	4.36ns	0.023	1.64 KB
#6521	`EnrichedLog`	netcoreapp3.1	2.14μs	0.873ns	3.38ns	0.0216	1.64 KB
#6521	`EnrichedLog`	net472	2.78μs	1.14ns	4.43ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	118μs	185ns	716ns	0.0579	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	121μs	234ns	906ns	0	0	4.28 KB
master	`EnrichedLog`	net472	151μs	111ns	416ns	0.683	0.228	4.46 KB
#6521	`EnrichedLog`	net6.0	117μs	236ns	913ns	0.0579	0	4.28 KB
#6521	`EnrichedLog`	netcoreapp3.1	122μs	239ns	927ns	0	0	4.28 KB
#6521	`EnrichedLog`	net472	152μs	129ns	498ns	0.686	0.229	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.07μs	1.38ns	5.34ns	0.0309	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.1μs	1.52ns	5.88ns	0.0284	2.2 KB
master	`EnrichedLog`	net472	4.85μs	1.48ns	5.74ns	0.32	2.02 KB
#6521	`EnrichedLog`	net6.0	2.91μs	1.25ns	4.86ns	0.0306	2.2 KB
#6521	`EnrichedLog`	netcoreapp3.1	4.3μs	1.67ns	6.26ns	0.0282	2.2 KB
#6521	`EnrichedLog`	net472	5.13μs	10.9ns	42.3ns	0.32	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.45μs	0.662ns	2.56ns	0.0159	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.75μs	0.776ns	3ns	0.0149	1.14 KB
master	`SendReceive`	net472	2.2μs	1.96ns	7.34ns	0.183	1.16 KB
#6521	`SendReceive`	net6.0	1.41μs	0.6ns	2.32ns	0.0161	1.14 KB
#6521	`SendReceive`	netcoreapp3.1	1.71μs	1.11ns	4.17ns	0.0154	1.14 KB
#6521	`SendReceive`	net472	2.11μs	2.59ns	10ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.72μs	0.639ns	2.39ns	0.022	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.95μs	2.64ns	10.2ns	0.0218	1.65 KB
master	`EnrichedLog`	net472	4.38μs	2.57ns	9.62ns	0.322	2.04 KB
#6521	`EnrichedLog`	net6.0	2.78μs	3.62ns	14ns	0.0221	1.6 KB
#6521	`EnrichedLog`	netcoreapp3.1	3.83μs	1.14ns	4.4ns	0.021	1.65 KB
#6521	`EnrichedLog`	net472	4.44μs	4.05ns	15.7ns	0.323	2.04 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6521

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0	1.158	405.15	469.23
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.155	480.19	554.73

Faster 🎉 in #6521

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑netcoreapp3.1	1.141	630.48	552.69

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	405ns	0.229ns	0.887ns	0.00811	576 B
master	`StartFinishSpan`	netcoreapp3.1	627ns	1.82ns	7.05ns	0.00787	576 B
master	`StartFinishSpan`	net472	668ns	0.476ns	1.84ns	0.0915	578 B
master	`StartFinishScope`	net6.0	480ns	0.414ns	1.6ns	0.00975	696 B
master	`StartFinishScope`	netcoreapp3.1	747ns	0.676ns	2.62ns	0.00951	696 B
master	`StartFinishScope`	net472	899ns	0.539ns	2.09ns	0.105	658 B
#6521	`StartFinishSpan`	net6.0	469ns	0.262ns	0.98ns	0.00795	576 B
#6521	`StartFinishSpan`	netcoreapp3.1	552ns	0.609ns	2.28ns	0.00774	576 B
#6521	`StartFinishSpan`	net472	713ns	0.274ns	1.06ns	0.0915	578 B
#6521	`StartFinishScope`	net6.0	555ns	0.218ns	0.816ns	0.00975	696 B
#6521	`StartFinishScope`	netcoreapp3.1	711ns	0.529ns	2.05ns	0.0095	696 B
#6521	`StartFinishScope`	net472	862ns	0.521ns	1.88ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	707ns	0.623ns	2.41ns	0.00991	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	896ns	0.904ns	3.5ns	0.00945	696 B
master	`RunOnMethodBegin`	net472	1.07μs	0.802ns	3.11ns	0.104	658 B
#6521	`RunOnMethodBegin`	net6.0	683ns	0.631ns	2.44ns	0.00986	696 B
#6521	`RunOnMethodBegin`	netcoreapp3.1	895ns	0.702ns	2.53ns	0.00912	696 B
#6521	`RunOnMethodBegin`	net472	1.15μs	0.404ns	1.4ns	0.104	658 B

andrewlock · 2025-01-10T11:21:59Z

Yeah, that didn't work at all

## Summary of changes Attempt to fix flakiness in Azure Functions multi-sample build ## Reason for change #6472 added multi-version building for Azure Functions samples. Unfortunately, this caused the build stage to be flaky, because Azure Functions apparently generates and builds a [completely separate _.csproj_](Azure/azure-functions-dotnet-worker#1252). Unfortunately, it also ignores all the artifact output stuff, which means when we restore/build multiple versions of the app (with different api versions, it stomps over itself. e.g. we can see the sample app build itself doing the right thing: ``` 08:59:05 [DBG] Samples.AzureFunctions.V4Isolated -> /project/artifacts/bin/Samples.AzureFunctions.V4Isolated/release_net6.0_1.6.0/Samples.AzureFunctions.V4Isolated.dll 08:59:06 [DBG] Samples.AzureFunctions.V4Isolated -> /project/artifacts/publish/Samples.AzureFunctions.V4Isolated/release_net6.0_1.6.0/ ``` But there's also the "WorkerExtensions" thing which is going completely off the rails, ignores all of the Directory.Build.props etc and just does its own thing ``` 08:59:07 [DBG] WorkerExtensions -> /tmp/oh4iajvh.kcj/publishout/ 08:59:07 [DBG] WorkerExtensions -> /tmp/zv30zzl2.cxz/publishout/Microsoft.Azure.Functions.Worker.Extensions.dll 08:59:08 [DBG] WorkerExtensions -> /tmp/zv30zzl2.cxz/publishout/ ``` Unfortunately, this shows it's trying to build two separate versions to the same location, and things break. ## Implementation details My initial attempt in #6521 failed. In this attempt I split the v1 SDK and v2 SDK into two separate projects. The SDK versions are tracked separately. I was thinking that as long as we only build a _single_ version per project we _should_ be ok. It was not OK. So in the end I tore out the Azure Functions version tracking. We could/should consider adding it back in some way, but right now this is causing too many issues An alternative is to just remove the Azure Functions samples from the "auto-updating multi version". I kept the split between v1 and v2 of the Azure Functions SDK to give us a _little_ more coverage; I think it's about the best we can do. ## Test coverage If this all finally passes, we should be good. ## Other details An important thing to note is that _currently_, even though we will now get dependabot notifactions about updates to the azure functions libs, these _aren't_ tested automatically. That's because currently our testing on Windows does _not_ run the "multi version" tests like we do on Linux. And we _only_ test Azure functions on Windows. This is something I'd like to address longer term with PRs like #6498, but for the meantime, we'll need to manually keep the samples up to date with the latest SDK version when there's a dependabot PR

andrewlock added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Jan 9, 2025

github-actions bot added the area:tests unit tests, integration tests label Jan 9, 2025

kevingosse approved these changes Jan 9, 2025

View reviewed changes

andrewlock force-pushed the andrew/try-fix-azure-functions-build branch from c737f86 to fe035e1 Compare January 9, 2025 11:02

andrewlock force-pushed the andrew/try-fix-azure-functions-build branch from fe035e1 to b5402cd Compare January 9, 2025 11:59

Fix it all hopefully

a1c5f0b

andrewlock force-pushed the andrew/try-fix-azure-functions-build branch from 476430e to a1c5f0b Compare January 9, 2025 12:47

bouwkast approved these changes Jan 9, 2025

View reviewed changes

andrewlock mentioned this pull request Jan 10, 2025

Fix flakiness in Azure Functions #6524

Merged

andrewlock closed this Jan 10, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Attempt to fix flakiness in Azure Functions multi-sample build #6521

Attempt to fix flakiness in Azure Functions multi-sample build #6521

andrewlock commented Jan 9, 2025 •

edited

Loading

datadog-ddstaging bot commented Jan 9, 2025 •

edited

Loading

andrewlock commented Jan 9, 2025

andrewlock commented Jan 10, 2025

andrewlock commented Jan 10, 2025

Raw results

Raw results

Raw results

More allocations ⚠️ in #6521

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6521

Faster 🎉 in #6521

Raw results

Raw results

andrewlock commented Jan 10, 2025

Attempt to fix flakiness in Azure Functions multi-sample build #6521

Attempt to fix flakiness in Azure Functions multi-sample build #6521

Conversation

andrewlock commented Jan 9, 2025 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

datadog-ddstaging bot commented Jan 9, 2025 • edited Loading

Datadog Report

⌛ Performance Regressions vs Default Branch (1)

andrewlock commented Jan 9, 2025

Execution-Time Benchmarks Report ⏱️

andrewlock commented Jan 10, 2025

Throughput/Crank Report ⚡

andrewlock commented Jan 10, 2025

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

More allocations ⚠️ in #6521

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6521

Faster 🎉 in #6521

Raw results

Raw results

andrewlock commented Jan 10, 2025

andrewlock commented Jan 9, 2025 •

edited

Loading

datadog-ddstaging bot commented Jan 9, 2025 •

edited

Loading