Parallelise system tests in CI #6496

e-n-0 · 2025-01-02T15:56:10Z

Summary of changes

Parallelise all system-tests scenarios in the CI.
All group scenarios have been broke down to scenarios, and each individual scenario are run in a specific job.
Remove old scenario LIBRARY_CONF_CUSTOM_HEADERS_SHORT and LIBRARY_CONF_CUSTOM_HEADERS_LONG from the running list. They are both now linked to the LIBRARY_CONF_CUSTOM_HEADER_TAGS scenario (which was added to the additionalScenarios list)

The majority of all the scenarios get completed in less than 5 min.
From 1h before the parallelisation:

To a bit more than 45 min after (depending of the attribution of runners). That's also the parametric tests that are taking now the more time:

Implementation details

List all scenarios that are part of groups

This is using the following command to get the scenarios from a scenario group name:
PYTHONPATH=. python utils/scripts/compute-workflow-parameters.py dotnet -g appsec

This way we can dynamically get the available weblogs in the system-tests and all scenarios.

Build the matrix

In order to create the matrix, a python script have been created to merge all scenarios for listed groups and the additional scenario that we want to run that are not part of a group.

We grep the output of the system-test command to fully get all scenarios and construct back a json dictionary with all the scenario name and the weblog variant that need to be used with it.

Update

For future update, if a new group scenario is added in the system-tests, it should be added to this variable list:
scenarioGroups: "essentials,parametric,appsec,remote-config,telemetry,integrations,debugger,profiling"

Build time

The weblog and agent docker images are build before and given as artefacts for all test jobs to reduce the completion time.

Reason for change

The system-tests were timing out in a bunch of PRs. Some group scenarios took more than 1h to be run.

datadog-ddstaging · 2025-01-02T16:26:28Z

Datadog Report

Branch report: flavien/asm/systemtests-timeout
Commit report: 734cfd7
Test service: dd-trace-dotnet

✅ 0 Failed, 234047 Passed, 2085 Skipped, 18h 31m 9.78s Total Time

andrewlock · 2025-01-02T16:50:57Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6496) - mean (68ms)  : 66, 71
     .   : milestone, 68,
    master - mean (68ms)  : 65, 72
     .   : milestone, 68,

    section CallTarget+Inlining+NGEN
    This PR (6496) - mean (978ms)  : 945, 1011
     .   : milestone, 978,
    master - mean (971ms)  : 949, 993
     .   : milestone, 971,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6496) - mean (107ms)  : 105, 110
     .   : milestone, 107,
    master - mean (107ms)  : 104, 109
     .   : milestone, 107,

    section CallTarget+Inlining+NGEN
    This PR (6496) - mean (673ms)  : 660, 685
     .   : milestone, 673,
    master - mean (674ms)  : 658, 690
     .   : milestone, 674,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6496) - mean (91ms)  : 89, 92
     .   : milestone, 91,
    master - mean (90ms)  : 88, 92
     .   : milestone, 90,

    section CallTarget+Inlining+NGEN
    This PR (6496) - mean (626ms)  : 611, 641
     .   : milestone, 626,
    master - mean (631ms)  : 615, 648
     .   : milestone, 631,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6496) - mean (194ms)  : 190, 198
     .   : milestone, 194,
    master - mean (194ms)  : 189, 198
     .   : milestone, 194,

    section CallTarget+Inlining+NGEN
    This PR (6496) - mean (1,104ms)  : 1075, 1133
     .   : milestone, 1104,
    master - mean (1,102ms)  : 1065, 1139
     .   : milestone, 1102,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6496) - mean (279ms)  : 275, 283
     .   : milestone, 279,
    master - mean (279ms)  : 275, 283
     .   : milestone, 279,

    section CallTarget+Inlining+NGEN
    This PR (6496) - mean (870ms)  : 838, 902
     .   : milestone, 870,
    master - mean (870ms)  : 844, 896
     .   : milestone, 870,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6496) - mean (266ms)  : 262, 271
     .   : milestone, 266,
    master - mean (268ms)  : 263, 272
     .   : milestone, 268,

    section CallTarget+Inlining+NGEN
    This PR (6496) - mean (847ms)  : 806, 887
     .   : milestone, 847,
    master - mean (855ms)  : 820, 890
     .   : milestone, 855,

andrewlock · 2025-01-02T17:45:23Z

Benchmarks Report for tracer 🐌

Benchmarks for #6496 compared to master:

All benchmarks have the same speed
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	7.99μs	44.6ns	303ns	0.016	0.00399	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.3μs	55.8ns	325ns	0.0152	0.00507	0	5.8 KB
master	`StartStopWithChild`	net472	16.4μs	68.2ns	264ns	1.04	0.307	0.0914	6.2 KB
#6496	`StartStopWithChild`	net6.0	7.81μs	40.5ns	246ns	0.0115	0.00385	0	5.61 KB
#6496	`StartStopWithChild`	netcoreapp3.1	10μs	52ns	260ns	0.0217	0.0108	0	5.8 KB
#6496	`StartStopWithChild`	net472	16.5μs	41.9ns	157ns	1.04	0.292	0.0973	6.21 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	503μs	624ns	2.41μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	660μs	515ns	2μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	858μs	757ns	2.93μs	0.431	3.3 KB
#6496	`WriteAndFlushEnrichedTraces`	net6.0	480μs	187ns	701ns	0	2.7 KB
#6496	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	646μs	417ns	1.56μs	0	2.7 KB
#6496	`WriteAndFlushEnrichedTraces`	net472	855μs	410ns	1.59μs	0.425	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	154μs	905ns	8.24μs	0.145	14.47 KB
master	`SendRequest`	netcoreapp3.1	177μs	1.02μs	7.96μs	0.163	17.27 KB
master	`SendRequest`	net472	0.00113ns	0.000522ns	0.00202ns	0	0 b
#6496	`SendRequest`	net6.0	156μs	866ns	7.35μs	0.145	14.47 KB
#6496	`SendRequest`	netcoreapp3.1	173μs	955ns	8.49μs	0.169	17.27 KB
#6496	`SendRequest`	net472	0.000322ns	0.000322ns	0.00116ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	571μs	3.23μs	24.4μs	0.543	0	0	41.6 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	662μs	3.78μs	27μs	0.391	0	0	41.71 KB
master	`WriteAndFlushEnrichedTraces`	net472	852μs	3.98μs	15.4μs	8.39	2.52	0.419	53.33 KB
#6496	`WriteAndFlushEnrichedTraces`	net6.0	581μs	3.31μs	23.1μs	0.558	0	0	41.79 KB
#6496	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	697μs	3.71μs	18.5μs	0.347	0	0	41.63 KB
#6496	`WriteAndFlushEnrichedTraces`	net472	854μs	4.13μs	17μs	8.19	2.59	0.431	53.27 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.2μs	1.4ns	5.43ns	0.0144	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.77μs	1.29ns	4.99ns	0.0132	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.1μs	3.38ns	13.1ns	0.157	0.00105	987 B
#6496	`ExecuteNonQuery`	net6.0	1.29μs	0.924ns	3.46ns	0.0142	0	1.02 KB
#6496	`ExecuteNonQuery`	netcoreapp3.1	1.85μs	2.17ns	8.39ns	0.0136	0	1.02 KB
#6496	`ExecuteNonQuery`	net472	2.13μs	1.88ns	7.26ns	0.156	0.00107	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.22μs	1.79ns	6.92ns	0.014	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.5μs	2.37ns	8.21ns	0.0129	976 B
master	`CallElasticsearch`	net472	2.59μs	2.4ns	9.29ns	0.158	995 B
master	`CallElasticsearchAsync`	net6.0	1.35μs	0.501ns	1.81ns	0.0135	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.64μs	1.03ns	3.97ns	0.014	1.02 KB
master	`CallElasticsearchAsync`	net472	2.57μs	2.36ns	9.13ns	0.167	1.05 KB
#6496	`CallElasticsearch`	net6.0	1.29μs	1.4ns	5.43ns	0.0134	976 B
#6496	`CallElasticsearch`	netcoreapp3.1	1.52μs	0.476ns	1.78ns	0.0128	976 B
#6496	`CallElasticsearch`	net472	2.53μs	3.5ns	13.5ns	0.158	995 B
#6496	`CallElasticsearchAsync`	net6.0	1.32μs	1.25ns	4.69ns	0.013	952 B
#6496	`CallElasticsearchAsync`	netcoreapp3.1	1.61μs	0.569ns	2.05ns	0.0141	1.02 KB
#6496	`CallElasticsearchAsync`	net472	2.71μs	0.959ns	3.71ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.37μs	0.772ns	2.89ns	0.013	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.65μs	1.19ns	4.44ns	0.0124	952 B
master	`ExecuteAsync`	net472	1.77μs	0.331ns	1.24ns	0.145	915 B
#6496	`ExecuteAsync`	net6.0	1.39μs	0.859ns	3.21ns	0.0132	952 B
#6496	`ExecuteAsync`	netcoreapp3.1	1.54μs	0.571ns	2.06ns	0.0124	952 B
#6496	`ExecuteAsync`	net472	1.81μs	0.409ns	1.53ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.52μs	1.57ns	5.88ns	0.0316	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.2μs	2.32ns	9ns	0.0388	2.85 KB
master	`SendAsync`	net472	7.47μs	1.93ns	7.47ns	0.496	3.12 KB
#6496	`SendAsync`	net6.0	4.47μs	2.1ns	7.87ns	0.0314	2.31 KB
#6496	`SendAsync`	netcoreapp3.1	5.34μs	5.24ns	19.6ns	0.0377	2.85 KB
#6496	`SendAsync`	net472	7.34μs	1.79ns	6.46ns	0.493	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.47μs	0.695ns	2.6ns	0.023	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.23μs	1.01ns	3.78ns	0.0221	1.64 KB
master	`EnrichedLog`	net472	2.82μs	1.59ns	5.93ns	0.249	1.57 KB
#6496	`EnrichedLog`	net6.0	1.57μs	0.948ns	3.55ns	0.0234	1.64 KB
#6496	`EnrichedLog`	netcoreapp3.1	2.12μs	0.482ns	1.74ns	0.0225	1.64 KB
#6496	`EnrichedLog`	net472	2.78μs	2.33ns	8.7ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	117μs	110ns	425ns	0.0582	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	121μs	250ns	968ns	0	0	4.28 KB
master	`EnrichedLog`	net472	152μs	106ns	398ns	0.683	0.228	4.46 KB
#6496	`EnrichedLog`	net6.0	117μs	245ns	949ns	0.0582	0	4.28 KB
#6496	`EnrichedLog`	netcoreapp3.1	121μs	178ns	691ns	0	0	4.28 KB
#6496	`EnrichedLog`	net472	151μs	226ns	875ns	0.686	0.229	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.04μs	1.18ns	4.4ns	0.0299	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.26μs	2.26ns	8.77ns	0.0298	2.2 KB
master	`EnrichedLog`	net472	4.9μs	2.9ns	10.8ns	0.319	2.02 KB
#6496	`EnrichedLog`	net6.0	2.97μs	2.88ns	10.8ns	0.0302	2.2 KB
#6496	`EnrichedLog`	netcoreapp3.1	4.25μs	3.29ns	12.8ns	0.0277	2.2 KB
#6496	`EnrichedLog`	net472	4.97μs	1.07ns	4.14ns	0.319	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.31μs	0.763ns	2.86ns	0.0164	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.82μs	0.67ns	2.42ns	0.0154	1.14 KB
master	`SendReceive`	net472	2.1μs	1.22ns	4.71ns	0.183	1.16 KB
#6496	`SendReceive`	net6.0	1.3μs	1.55ns	5.8ns	0.0158	1.14 KB
#6496	`SendReceive`	netcoreapp3.1	1.81μs	1.15ns	4.45ns	0.0156	1.14 KB
#6496	`SendReceive`	net472	2.04μs	0.737ns	2.66ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.72μs	0.802ns	3ns	0.0218	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.98μs	1.3ns	5.05ns	0.0217	1.65 KB
master	`EnrichedLog`	net472	4.39μs	3.91ns	15.1ns	0.322	2.04 KB
#6496	`EnrichedLog`	net6.0	2.75μs	1.1ns	3.97ns	0.022	1.6 KB
#6496	`EnrichedLog`	netcoreapp3.1	3.97μs	3.76ns	14.5ns	0.0216	1.65 KB
#6496	`EnrichedLog`	net472	4.3μs	5.27ns	20.4ns	0.323	2.04 KB

Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	409ns	0.257ns	0.995ns	0.00802	576 B
master	`StartFinishSpan`	netcoreapp3.1	564ns	1.45ns	5.61ns	0.00768	576 B
master	`StartFinishSpan`	net472	636ns	0.721ns	2.79ns	0.0917	578 B
master	`StartFinishScope`	net6.0	549ns	0.265ns	0.993ns	0.00987	696 B
master	`StartFinishScope`	netcoreapp3.1	681ns	0.888ns	3.44ns	0.00929	696 B
master	`StartFinishScope`	net472	901ns	1.03ns	3.97ns	0.104	658 B
#6496	`StartFinishSpan`	net6.0	405ns	0.173ns	0.672ns	0.00809	576 B
#6496	`StartFinishSpan`	netcoreapp3.1	564ns	2.68ns	10.4ns	0.00772	576 B
#6496	`StartFinishSpan`	net472	705ns	0.476ns	1.84ns	0.0918	578 B
#6496	`StartFinishScope`	net6.0	563ns	0.211ns	0.788ns	0.00968	696 B
#6496	`StartFinishScope`	netcoreapp3.1	685ns	0.563ns	2.18ns	0.00923	696 B
#6496	`StartFinishScope`	net472	860ns	0.47ns	1.82ns	0.105	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	656ns	0.399ns	1.54ns	0.00976	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	932ns	0.652ns	2.53ns	0.00932	696 B
master	`RunOnMethodBegin`	net472	1.11μs	0.355ns	1.33ns	0.104	658 B
#6496	`RunOnMethodBegin`	net6.0	688ns	0.182ns	0.707ns	0.00973	696 B
#6496	`RunOnMethodBegin`	netcoreapp3.1	972ns	0.968ns	3.49ns	0.00943	696 B
#6496	`RunOnMethodBegin`	net472	1.08μs	0.575ns	2.23ns	0.104	658 B

andrewlock · 2025-01-06T17:06:40Z

Throughput/Crank Report ⚡

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6496) (11.094M)   : 0, 11093830
    master (11.068M)   : 0, 11067812
    benchmarks/2.9.0 (11.045M)   : 0, 11045405

    section Automatic
    This PR (6496) (7.236M)   : 0, 7235958
    master (7.170M)   : 0, 7169979
    benchmarks/2.9.0 (7.885M)   : 0, 7885346

    section Trace stats
    master (7.672M)   : 0, 7671870

    section Manual
    master (11.135M)   : 0, 11135060

    section Manual + Automatic
    This PR (6496) (6.683M)   : 0, 6682530
    master (6.686M)   : 0, 6686295

    section DD_TRACE_ENABLED=0
    master (10.301M)   : 0, 10301221

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6496) (9.655M)   : 0, 9655389
    master (9.415M)   : 0, 9414830
    benchmarks/2.9.0 (9.586M)   : 0, 9586476

    section Automatic
    This PR (6496) (6.436M)   : 0, 6436042
    master (6.138M)   : 0, 6138011

    section Trace stats
    master (6.597M)   : 0, 6596896

    section Manual
    master (9.454M)   : 0, 9454016

    section Manual + Automatic
    This PR (6496) (5.871M)   : 0, 5870860
    master (5.945M)   : 0, 5944630

    section DD_TRACE_ENABLED=0
    master (8.978M)   : 0, 8978445

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6496) (9.953M)   : 0, 9952961
    master (10.157M)   : 0, 10156614

    section Automatic
    This PR (6496) (6.610M)   : 0, 6609732
    master (6.561M)   : 0, 6560916

    section Trace stats
    master (7.244M)   : 0, 7243820

    section Manual
    master (9.958M)   : 0, 9958235

    section Manual + Automatic
    This PR (6496) (6.394M)   : 0, 6393983
    master (6.193M)   : 0, 6192782

    section DD_TRACE_ENABLED=0
    master (9.470M)   : 0, 9470030

e-n-0 requested a review from a team as a code owner January 2, 2025 15:56

e-n-0 marked this pull request as draft January 2, 2025 15:56

github-actions bot added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Jan 2, 2025

Increase timeout to 75 min

999d88a

e-n-0 force-pushed the flavien/asm/systemtests-timeout branch from 5e046e6 to 999d88a Compare January 2, 2025 16:00

e-n-0 marked this pull request as ready for review January 2, 2025 16:01

e-n-0 marked this pull request as draft January 2, 2025 16:18

revert time

a3e4dd7

e-n-0 removed the area:integrations label Jan 3, 2025

e-n-0 force-pushed the flavien/asm/systemtests-timeout branch 4 times, most recently from a4888af to dd65986 Compare January 3, 2025 14:20

e-n-0 changed the title ~~Increase system tests timeout to 75 min~~ Parallelise system tests in CI Jan 3, 2025

jobs matrix run scenarios

0939a7a

e-n-0 force-pushed the flavien/asm/systemtests-timeout branch 6 times, most recently from 57cccf3 to 078a37d Compare January 4, 2025 13:21

fix

5efa28b

e-n-0 force-pushed the flavien/asm/systemtests-timeout branch 2 times, most recently from 8cd9e80 to b80f73e Compare January 4, 2025 20:17

Change name of job

8a73064

e-n-0 force-pushed the flavien/asm/systemtests-timeout branch from b80f73e to 8a73064 Compare January 4, 2025 20:19

move build into prior job

187c2d4

e-n-0 force-pushed the flavien/asm/systemtests-timeout branch from 021d0b5 to 0e2c947 Compare January 6, 2025 12:27

update to build all weblogs and load the correct one in the scenario

3dbd068

e-n-0 force-pushed the flavien/asm/systemtests-timeout branch from 0e2c947 to 3dbd068 Compare January 6, 2025 13:13

only publish the artefacts once and not on retry

734cfd7

e-n-0 force-pushed the flavien/asm/systemtests-timeout branch from 70888cb to 734cfd7 Compare January 6, 2025 15:37

e-n-0 marked this pull request as ready for review January 6, 2025 17:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parallelise system tests in CI #6496

Parallelise system tests in CI #6496

e-n-0 commented Jan 2, 2025 •

edited

Loading

datadog-ddstaging bot commented Jan 2, 2025 •

edited

Loading

andrewlock commented Jan 2, 2025 •

edited

Loading

andrewlock commented Jan 2, 2025 •

edited

Loading

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

andrewlock commented Jan 6, 2025

Parallelise system tests in CI #6496

Are you sure you want to change the base?

Parallelise system tests in CI #6496

Conversation

e-n-0 commented Jan 2, 2025 • edited Loading

Summary of changes

Implementation details

List all scenarios that are part of groups

Build the matrix

Update

Build time

Reason for change

datadog-ddstaging bot commented Jan 2, 2025 • edited Loading

Datadog Report

andrewlock commented Jan 2, 2025 • edited Loading

Execution-Time Benchmarks Report ⏱️

andrewlock commented Jan 2, 2025 • edited Loading

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

andrewlock commented Jan 6, 2025

Throughput/Crank Report ⚡

e-n-0 commented Jan 2, 2025 •

edited

Loading

datadog-ddstaging bot commented Jan 2, 2025 •

edited

Loading

andrewlock commented Jan 2, 2025 •

edited

Loading

andrewlock commented Jan 2, 2025 •

edited

Loading