Implement tests quarantining #8320

nikita-tkachenko-datadog · 2025-01-31T13:08:17Z

What Does This Do

Implements tests quarantining.
A quarantined test case is run as usual, but if it fails the status of the build is not affected (i.e. if a quarantined test fails, the build can still be successful).

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: SDTEST-1483

pr-commenter · 2025-01-31T13:50:19Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/test-quarantine
git_commit_date	1738318728	1738329441
git_commit_sha	`b24d153`	`6c76dc4`
release_version	1.47.0-SNAPSHOT~b24d15319f	1.47.0-SNAPSHOT~6c76dc4a0f

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1738332058	1738332058
ci_job_id	789163391	789163391
ci_pipeline_id	54567933	54567933
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-5p9ten-project-304-concurrent-0-i2pykog3 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-5p9ten-project-304-concurrent-0-i2pykog3 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 53 metrics, 10 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.47.0-SNAPSHOT~6c76dc4a0f, baseline=1.47.0-SNAPSHOT~b24d15319f

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.041 s) : 0, 1041487
Total [baseline] (8.65 s) : 0, 8650489
Agent [candidate] (1.039 s) : 0, 1039324
Total [candidate] (8.624 s) : 0, 8623785
section iast
Agent [baseline] (1.171 s) : 0, 1170874
Total [baseline] (9.195 s) : 0, 9194553
Agent [candidate] (1.17 s) : 0, 1169782
Total [candidate] (9.211 s) : 0, 9210981
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.177 s) : 0, 1177012
Total [baseline] (9.174 s) : 0, 9173702
Agent [candidate] (1.169 s) : 0, 1168727
Total [candidate] (9.17 s) : 0, 9169658
section iast_TELEMETRY_OFF
Agent [baseline] (1.163 s) : 0, 1163261
Total [baseline] (9.187 s) : 0, 9186727
Agent [candidate] (1.167 s) : 0, 1166532
Total [candidate] (9.201 s) : 0, 9200586

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.041 s	-
Agent	iast	1.171 s	129.386 ms (12.4%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.177 s	135.524 ms (13.0%)
Agent	iast_TELEMETRY_OFF	1.163 s	121.774 ms (11.7%)
Total	tracing	8.65 s	-
Total	iast	9.195 s	544.063 ms (6.3%)
Total	iast_HARDCODED_SECRET_DISABLED	9.174 s	523.213 ms (6.0%)
Total	iast_TELEMETRY_OFF	9.187 s	536.238 ms (6.2%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.039 s	-
Agent	iast	1.17 s	130.459 ms (12.6%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.169 s	129.403 ms (12.5%)
Agent	iast_TELEMETRY_OFF	1.167 s	127.209 ms (12.2%)
Total	tracing	8.624 s	-
Total	iast	9.211 s	587.197 ms (6.8%)
Total	iast_HARDCODED_SECRET_DISABLED	9.17 s	545.873 ms (6.3%)
Total	iast_TELEMETRY_OFF	9.201 s	576.801 ms (6.7%)

gantt
    title insecure-bank - break down per module: candidate=1.47.0-SNAPSHOT~6c76dc4a0f, baseline=1.47.0-SNAPSHOT~b24d15319f

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (717.909 ms) : 0, 717909
BytebuddyAgent [candidate] (716.864 ms) : 0, 716864
GlobalTracer [baseline] (241.109 ms) : 0, 241109
GlobalTracer [candidate] (239.666 ms) : 0, 239666
AppSec [baseline] (55.297 ms) : 0, 55297
AppSec [candidate] (55.361 ms) : 0, 55361
Remote Config [baseline] (704.758 µs) : 0, 705
Remote Config [candidate] (721.792 µs) : 0, 722
Telemetry [baseline] (11.244 ms) : 0, 11244
Telemetry [candidate] (11.435 ms) : 0, 11435
section iast
BytebuddyAgent [baseline] (834.627 ms) : 0, 834627
BytebuddyAgent [candidate] (834.068 ms) : 0, 834068
GlobalTracer [baseline] (231.847 ms) : 0, 231847
GlobalTracer [candidate] (231.798 ms) : 0, 231798
AppSec [baseline] (55.54 ms) : 0, 55540
AppSec [candidate] (53.053 ms) : 0, 53053
Remote Config [baseline] (618.779 µs) : 0, 619
Remote Config [candidate] (604.708 µs) : 0, 605
Telemetry [baseline] (8.73 ms) : 0, 8730
Telemetry [candidate] (8.688 ms) : 0, 8688
IAST [baseline] (24.218 ms) : 0, 24218
IAST [candidate] (26.309 ms) : 0, 26309
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (839.711 ms) : 0, 839711
BytebuddyAgent [candidate] (833.427 ms) : 0, 833427
GlobalTracer [baseline] (232.341 ms) : 0, 232341
GlobalTracer [candidate] (230.922 ms) : 0, 230922
AppSec [baseline] (54.106 ms) : 0, 54106
AppSec [candidate] (54.687 ms) : 0, 54687
Remote Config [baseline] (618.862 µs) : 0, 619
Remote Config [candidate] (610.69 µs) : 0, 611
Telemetry [baseline] (8.751 ms) : 0, 8751
Telemetry [candidate] (8.669 ms) : 0, 8669
IAST [baseline] (25.916 ms) : 0, 25916
IAST [candidate] (25.128 ms) : 0, 25128
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (829.251 ms) : 0, 829251
BytebuddyAgent [candidate] (831.655 ms) : 0, 831655
GlobalTracer [baseline] (230.43 ms) : 0, 230430
GlobalTracer [candidate] (231.276 ms) : 0, 231276
AppSec [baseline] (53.822 ms) : 0, 53822
AppSec [candidate] (54.604 ms) : 0, 54604
Remote Config [baseline] (617.171 µs) : 0, 617
Remote Config [candidate] (613.443 µs) : 0, 613
Telemetry [baseline] (8.615 ms) : 0, 8615
Telemetry [candidate] (8.54 ms) : 0, 8540
IAST [baseline] (25.28 ms) : 0, 25280
IAST [candidate] (24.581 ms) : 0, 24581

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.47.0-SNAPSHOT~6c76dc4a0f, baseline=1.47.0-SNAPSHOT~b24d15319f

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.037 s) : 0, 1036686
Total [baseline] (10.432 s) : 0, 10431982
Agent [candidate] (1.042 s) : 0, 1042256
Total [candidate] (10.497 s) : 0, 10496830
section appsec
Agent [baseline] (1.186 s) : 0, 1185559
Total [baseline] (10.777 s) : 0, 10776729
Agent [candidate] (1.191 s) : 0, 1191195
Total [candidate] (10.765 s) : 0, 10764796
section iast
Agent [baseline] (1.17 s) : 0, 1169570
Total [baseline] (10.999 s) : 0, 10999237
Agent [candidate] (1.168 s) : 0, 1168267
Total [candidate] (10.919 s) : 0, 10919091
section profiling
Agent [baseline] (1.259 s) : 0, 1259307
Total [baseline] (10.886 s) : 0, 10885606
Agent [candidate] (1.258 s) : 0, 1257609
Total [candidate] (10.811 s) : 0, 10810724

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.037 s	-
Agent	appsec	1.186 s	148.872 ms (14.4%)
Agent	iast	1.17 s	132.883 ms (12.8%)
Agent	profiling	1.259 s	222.621 ms (21.5%)
Total	tracing	10.432 s	-
Total	appsec	10.777 s	344.747 ms (3.3%)
Total	iast	10.999 s	567.254 ms (5.4%)
Total	profiling	10.886 s	453.624 ms (4.3%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.042 s	-
Agent	appsec	1.191 s	148.939 ms (14.3%)
Agent	iast	1.168 s	126.011 ms (12.1%)
Agent	profiling	1.258 s	215.353 ms (20.7%)
Total	tracing	10.497 s	-
Total	appsec	10.765 s	267.967 ms (2.6%)
Total	iast	10.919 s	422.261 ms (4.0%)
Total	profiling	10.811 s	313.895 ms (3.0%)

gantt
    title petclinic - break down per module: candidate=1.47.0-SNAPSHOT~6c76dc4a0f, baseline=1.47.0-SNAPSHOT~b24d15319f

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (714.058 ms) : 0, 714058
BytebuddyAgent [candidate] (716.273 ms) : 0, 716273
GlobalTracer [baseline] (239.883 ms) : 0, 239883
GlobalTracer [candidate] (240.313 ms) : 0, 240313
AppSec [baseline] (55.264 ms) : 0, 55264
AppSec [candidate] (55.254 ms) : 0, 55254
Remote Config [baseline] (712.631 µs) : 0, 713
Remote Config [candidate] (711.727 µs) : 0, 712
Telemetry [baseline] (11.574 ms) : 0, 11574
Telemetry [candidate] (14.476 ms) : 0, 14476
section appsec
BytebuddyAgent [baseline] (735.38 ms) : 0, 735380
BytebuddyAgent [candidate] (738.996 ms) : 0, 738996
GlobalTracer [baseline] (238.561 ms) : 0, 238561
GlobalTracer [candidate] (239.709 ms) : 0, 239709
AppSec [baseline] (176.372 ms) : 0, 176372
AppSec [candidate] (177.188 ms) : 0, 177188
Remote Config [baseline] (651.554 µs) : 0, 652
Remote Config [candidate] (657.889 µs) : 0, 658
Telemetry [baseline] (8.295 ms) : 0, 8295
Telemetry [candidate] (8.383 ms) : 0, 8383
IAST [baseline] (22.047 ms) : 0, 22047
IAST [candidate] (21.762 ms) : 0, 21762
section iast
BytebuddyAgent [baseline] (834.046 ms) : 0, 834046
BytebuddyAgent [candidate] (833.16 ms) : 0, 833160
GlobalTracer [baseline] (231.186 ms) : 0, 231186
GlobalTracer [candidate] (231.156 ms) : 0, 231156
AppSec [baseline] (53.08 ms) : 0, 53080
AppSec [candidate] (53.712 ms) : 0, 53712
Remote Config [baseline] (606.093 µs) : 0, 606
Remote Config [candidate] (603.69 µs) : 0, 604
Telemetry [baseline] (8.661 ms) : 0, 8661
Telemetry [candidate] (8.704 ms) : 0, 8704
IAST [baseline] (26.696 ms) : 0, 26696
IAST [candidate] (25.664 ms) : 0, 25664
section profiling
BytebuddyAgent [baseline] (705.777 ms) : 0, 705777
BytebuddyAgent [candidate] (705.126 ms) : 0, 705126
GlobalTracer [baseline] (350.037 ms) : 0, 350037
GlobalTracer [candidate] (350.886 ms) : 0, 350886
AppSec [baseline] (55.981 ms) : 0, 55981
AppSec [candidate] (54.182 ms) : 0, 54182
Remote Config [baseline] (702.32 µs) : 0, 702
Remote Config [candidate] (699.735 µs) : 0, 700
Telemetry [baseline] (8.936 ms) : 0, 8936
Telemetry [candidate] (8.931 ms) : 0, 8931
ProfilingAgent [baseline] (95.551 ms) : 0, 95551
ProfilingAgent [candidate] (95.531 ms) : 0, 95531
Profiling [baseline] (95.576 ms) : 0, 95576
Profiling [candidate] (95.555 ms) : 0, 95555

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-01-31T13:30:59	2025-01-31T13:38:04
git_branch	master	nikita-tkachenko/test-quarantine
git_commit_date	1738318728	1738329441
git_commit_sha	`b24d153`	`6c76dc4`
release_version	1.47.0-SNAPSHOT~b24d15319f	1.47.0-SNAPSHOT~6c76dc4a0f
start_time	2025-01-31T13:30:45	2025-01-31T13:37:50

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1738331042	1738331042
ci_job_id	789163392	789163392
ci_pipeline_id	54567933	54567933
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-3e13i2t5-project-304-concurrent-0-2x4fb7td 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-3e13i2t5-project-304-concurrent-0-2x4fb7td 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 13 metrics, 15 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.47.0-SNAPSHOT~6c76dc4a0f, baseline=1.47.0-SNAPSHOT~b24d15319f
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.364 ms) : 1345, 1384
.   : milestone, 1364,
appsec (1.752 ms) : 1729, 1776
.   : milestone, 1752,
appsec_no_iast (1.77 ms) : 1748, 1793
.   : milestone, 1770,
iast (1.513 ms) : 1489, 1536
.   : milestone, 1513,
profiling (1.513 ms) : 1490, 1537
.   : milestone, 1513,
tracing (1.494 ms) : 1468, 1520
.   : milestone, 1494,
section candidate
no_agent (1.36 ms) : 1340, 1379
.   : milestone, 1360,
appsec (1.743 ms) : 1719, 1767
.   : milestone, 1743,
appsec_no_iast (1.748 ms) : 1723, 1773
.   : milestone, 1748,
iast (1.499 ms) : 1475, 1523
.   : milestone, 1499,
profiling (1.526 ms) : 1503, 1549
.   : milestone, 1526,
tracing (1.489 ms) : 1464, 1515
.   : milestone, 1489,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.364 ms [1.345 ms, 1.384 ms]	-
appsec	1.752 ms [1.729 ms, 1.776 ms]	387.96 µs (28.4%)
appsec_no_iast	1.77 ms [1.748 ms, 1.793 ms]	405.894 µs (29.8%)
iast	1.513 ms [1.489 ms, 1.536 ms]	148.276 µs (10.9%)
profiling	1.513 ms [1.49 ms, 1.537 ms]	149.054 µs (10.9%)
tracing	1.494 ms [1.468 ms, 1.52 ms]	129.475 µs (9.5%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.36 ms [1.34 ms, 1.379 ms]	-
appsec	1.743 ms [1.719 ms, 1.767 ms]	383.071 µs (28.2%)
appsec_no_iast	1.748 ms [1.723 ms, 1.773 ms]	388.64 µs (28.6%)
iast	1.499 ms [1.475 ms, 1.523 ms]	139.067 µs (10.2%)
profiling	1.526 ms [1.503 ms, 1.549 ms]	166.222 µs (12.2%)
tracing	1.489 ms [1.464 ms, 1.515 ms]	129.87 µs (9.6%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.47.0-SNAPSHOT~6c76dc4a0f, baseline=1.47.0-SNAPSHOT~b24d15319f
    dateFormat X
    axisFormat %s
section baseline
no_agent (389.992 µs) : 370, 410
.   : milestone, 390,
iast (521.526 µs) : 498, 545
.   : milestone, 522,
iast_FULL (750.216 µs) : 727, 773
.   : milestone, 750,
iast_GLOBAL (564.479 µs) : 541, 588
.   : milestone, 564,
iast_HARDCODED_SECRET_DISABLED (518.937 µs) : 497, 541
.   : milestone, 519,
iast_INACTIVE (463.058 µs) : 442, 484
.   : milestone, 463,
iast_TELEMETRY_OFF (499.602 µs) : 476, 523
.   : milestone, 500,
tracing (463.895 µs) : 442, 485
.   : milestone, 464,
section candidate
no_agent (381.511 µs) : 362, 401
.   : milestone, 382,
iast (510.476 µs) : 487, 534
.   : milestone, 510,
iast_FULL (750.15 µs) : 727, 774
.   : milestone, 750,
iast_GLOBAL (562.411 µs) : 540, 584
.   : milestone, 562,
iast_HARDCODED_SECRET_DISABLED (516.205 µs) : 494, 538
.   : milestone, 516,
iast_INACTIVE (463.781 µs) : 443, 485
.   : milestone, 464,
iast_TELEMETRY_OFF (498.039 µs) : 475, 521
.   : milestone, 498,
tracing (453.08 µs) : 433, 474
.   : milestone, 453,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	389.992 µs [369.989 µs, 409.995 µs]	-
iast	521.526 µs [498.283 µs, 544.769 µs]	131.534 µs (33.7%)
iast_FULL	750.216 µs [727.288 µs, 773.145 µs]	360.224 µs (92.4%)
iast_GLOBAL	564.479 µs [540.96 µs, 587.997 µs]	174.487 µs (44.7%)
iast_HARDCODED_SECRET_DISABLED	518.937 µs [497.216 µs, 540.658 µs]	128.945 µs (33.1%)
iast_INACTIVE	463.058 µs [441.713 µs, 484.403 µs]	73.066 µs (18.7%)
iast_TELEMETRY_OFF	499.602 µs [476.413 µs, 522.791 µs]	109.61 µs (28.1%)
tracing	463.895 µs [442.344 µs, 485.445 µs]	73.903 µs (18.9%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	381.511 µs [361.569 µs, 401.453 µs]	-
iast	510.476 µs [487.262 µs, 533.69 µs]	128.965 µs (33.8%)
iast_FULL	750.15 µs [726.6 µs, 773.701 µs]	368.639 µs (96.6%)
iast_GLOBAL	562.411 µs [540.353 µs, 584.468 µs]	180.9 µs (47.4%)
iast_HARDCODED_SECRET_DISABLED	516.205 µs [493.982 µs, 538.429 µs]	134.694 µs (35.3%)
iast_INACTIVE	463.781 µs [442.615 µs, 484.946 µs]	82.27 µs (21.6%)
iast_TELEMETRY_OFF	498.039 µs [474.964 µs, 521.113 µs]	116.528 µs (30.5%)
tracing	453.08 µs [432.631 µs, 473.529 µs]	71.569 µs (18.8%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/test-quarantine
git_commit_date	1738318728	1738329441
git_commit_sha	`b24d153`	`6c76dc4`
release_version	1.47.0-SNAPSHOT~b24d15319f	1.47.0-SNAPSHOT~6c76dc4a0f

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1738331496	1738331496
ci_job_id	789163393	789163393
ci_pipeline_id	54567933	54567933
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-5p9ten-project-304-concurrent-1-ubxukni9 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-5p9ten-project-304-concurrent-1-ubxukni9 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.47.0-SNAPSHOT~6c76dc4a0f, baseline=1.47.0-SNAPSHOT~b24d15319f
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.476 ms) : 1465, 1488
.   : milestone, 1476,
appsec (2.365 ms) : 2322, 2409
.   : milestone, 2365,
iast (2.109 ms) : 2053, 2164
.   : milestone, 2109,
iast_GLOBAL (2.157 ms) : 2101, 2212
.   : milestone, 2157,
profiling (1.994 ms) : 1949, 2039
.   : milestone, 1994,
tracing (1.953 ms) : 1911, 1995
.   : milestone, 1953,
section candidate
no_agent (1.469 ms) : 1458, 1481
.   : milestone, 1469,
appsec (2.369 ms) : 2326, 2413
.   : milestone, 2369,
iast (2.114 ms) : 2059, 2169
.   : milestone, 2114,
iast_GLOBAL (2.147 ms) : 2092, 2202
.   : milestone, 2147,
profiling (1.962 ms) : 1918, 2005
.   : milestone, 1962,
tracing (1.96 ms) : 1917, 2002
.   : milestone, 1960,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.476 ms [1.465 ms, 1.488 ms]	-
appsec	2.365 ms [2.322 ms, 2.409 ms]	889.047 µs (60.2%)
iast	2.109 ms [2.053 ms, 2.164 ms]	632.383 µs (42.8%)
iast_GLOBAL	2.157 ms [2.101 ms, 2.212 ms]	680.445 µs (46.1%)
profiling	1.994 ms [1.949 ms, 2.039 ms]	518.033 µs (35.1%)
tracing	1.953 ms [1.911 ms, 1.995 ms]	476.828 µs (32.3%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.469 ms [1.458 ms, 1.481 ms]	-
appsec	2.369 ms [2.326 ms, 2.413 ms]	900.122 µs (61.3%)
iast	2.114 ms [2.059 ms, 2.169 ms]	645.05 µs (43.9%)
iast_GLOBAL	2.147 ms [2.092 ms, 2.202 ms]	678.178 µs (46.2%)
profiling	1.962 ms [1.918 ms, 2.005 ms]	492.424 µs (33.5%)
tracing	1.96 ms [1.917 ms, 2.002 ms]	490.58 µs (33.4%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.47.0-SNAPSHOT~6c76dc4a0f, baseline=1.47.0-SNAPSHOT~b24d15319f
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.261 s) : 15261000, 15261000
.   : milestone, 15261000,
appsec (14.98 s) : 14980000, 14980000
.   : milestone, 14980000,
iast (18.632 s) : 18632000, 18632000
.   : milestone, 18632000,
iast_GLOBAL (17.995 s) : 17995000, 17995000
.   : milestone, 17995000,
profiling (15.217 s) : 15217000, 15217000
.   : milestone, 15217000,
tracing (15.212 s) : 15212000, 15212000
.   : milestone, 15212000,
section candidate
no_agent (14.956 s) : 14956000, 14956000
.   : milestone, 14956000,
appsec (15.02 s) : 15020000, 15020000
.   : milestone, 15020000,
iast (18.378 s) : 18378000, 18378000
.   : milestone, 18378000,
iast_GLOBAL (18.093 s) : 18093000, 18093000
.   : milestone, 18093000,
profiling (15.091 s) : 15091000, 15091000
.   : milestone, 15091000,
tracing (15.004 s) : 15004000, 15004000
.   : milestone, 15004000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.261 s [15.261 s, 15.261 s]	-
appsec	14.98 s [14.98 s, 14.98 s]	-281.0 ms (-1.8%)
iast	18.632 s [18.632 s, 18.632 s]	3.371 s (22.1%)
iast_GLOBAL	17.995 s [17.995 s, 17.995 s]	2.734 s (17.9%)
profiling	15.217 s [15.217 s, 15.217 s]	-44.0 ms (-0.3%)
tracing	15.212 s [15.212 s, 15.212 s]	-49.0 ms (-0.3%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.956 s [14.956 s, 14.956 s]	-
appsec	15.02 s [15.02 s, 15.02 s]	64.0 ms (0.4%)
iast	18.378 s [18.378 s, 18.378 s]	3.422 s (22.9%)
iast_GLOBAL	18.093 s [18.093 s, 18.093 s]	3.137 s (21.0%)
profiling	15.091 s [15.091 s, 15.091 s]	135.0 ms (0.9%)
tracing	15.004 s [15.004 s, 15.004 s]	48.0 ms (0.3%)

| Package | Type | Package file | Manager | Update | Change | |---|---|---|---|---|---| | [com.datadoghq:dd-trace-api](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.46.1` -> `1.47.0` | | [com.datadoghq:dd-trace-ot](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.46.1` -> `1.47.0` | | [software.amazon.awssdk:sdk-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | | [software.amazon.awssdk:sqs](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | | [software.amazon.awssdk:dynamodb-enhanced](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | | [software.amazon.awssdk:dynamodb](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | | [software.amazon.awssdk:aws-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | | [software.amazon.awssdk:bom](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | | [software.amazon.awssdk:auth](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.33` -> `2.30.34` | --- ### Release Notes <details> <summary>datadog/dd-trace-java (com.datadoghq:dd-trace-api)</summary> ### [`v1.47.0`](https://github.com/DataDog/dd-trace-java/releases/tag/v1.47.0): 1.47.0 ##### Components ##### Application Security Management (IAST) - 🐛 Exclude com.stripe.net.HttpURLConnectionClient to solve IAST SSRF vulnerability false positives ([#8483](DataDog/dd-trace-java#8483) - [@jandro996](https://github.com/jandro996)) - 🐛 Add exclusion to solve IAST weak randomness vulnerability false positives ([#8462](DataDog/dd-trace-java#8462) - [@jandro996](https://github.com/jandro996)) - ✨ Fix weak randomness false positive in Kafka client ([#8408](DataDog/dd-trace-java#8408) - [@smola](https://github.com/smola)) - ✨ Fix location for SSRF with Kong Unirest ([#8407](DataDog/dd-trace-java#8407) - [@smola](https://github.com/smola)) - ✨ Exclude IBM Instana from IAST ([#8406](DataDog/dd-trace-java#8406) - [@smola](https://github.com/smola)) - 🐛 Fix org.json iast instrumentation test for latest dependency ([#8347](DataDog/dd-trace-java#8347) - [@jandro996](https://github.com/jandro996)) - ✨ Configuration to Disable APM Tracing ([#8219](DataDog/dd-trace-java#8219) - [@jandro996](https://github.com/jandro996)) - ✨ Address cookie vulnerability cardinality issues ([#8210](DataDog/dd-trace-java#8210) - [@jandro996](https://github.com/jandro996)) - ✨ Email HTML Injection detection in IAST ([#8205](DataDog/dd-trace-java#8205) - [@sezen-datadog](https://github.com/sezen-datadog)) ##### Application Security Management (WAF) - 🐛✨ Ensure usr.exists tag is not overridden when UsernameNotFoundException is thrown ([#8376](DataDog/dd-trace-java#8376) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - 🐛✨ Ensure usr.exists tag is not overridden by auto instrumentation ([#8374](DataDog/dd-trace-java#8374) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Update appsec metrics with event_rules_version tag ([#8354](DataDog/dd-trace-java#8354) - [@sezen-datadog](https://github.com/sezen-datadog)) - ✨ Update metrics: appsec.waf.requests ([#8353](DataDog/dd-trace-java#8353) - [@Mariovido](https://github.com/Mariovido)) - ✨ Improve ASM support in vert.x 5.0 ([#8285](DataDog/dd-trace-java#8285) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Update metrics: appsec.waf.updates and appsec.waf.init ([#8280](DataDog/dd-trace-java#8280) - [@Mariovido](https://github.com/Mariovido)) - ✨ Configuration to Disable APM Tracing ([#8219](DataDog/dd-trace-java#8219) - [@jandro996](https://github.com/jandro996)) ##### Build & Tooling - 🐛 Do not generate Muzzle references for primitive arrays in method body ([#8361](DataDog/dd-trace-java#8361) - [@amarziali](https://github.com/amarziali)) - 📖 Improve dev env setup documentation for Windows ([#8180](DataDog/dd-trace-java#8180) - [@lucaspimentel](https://github.com/lucaspimentel)) ##### Continuous Integration Visibility - ✨ Add support for skip-EFD tagging ([#8487](DataDog/dd-trace-java#8487) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🐛 Fix an NPE in Gradle Android instrumentation ([#8484](DataDog/dd-trace-java#8484) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Consider modified tests when applying fail-fast tests ordering ([#8474](DataDog/dd-trace-java#8474) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Implement tests reordering for TestNG ([#8467](DataDog/dd-trace-java#8467) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🐛 Fix Gradle Launcher instrumentation to not interfere with Gradle Test Kit ([#8465](DataDog/dd-trace-java#8465) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🧹 Use separate TestEventHandlers per framework in CI Vis instrumentations ([#8451](DataDog/dd-trace-java#8451) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Remove warning log when JUnit 4 test method cannot be retrieved ([#8445](DataDog/dd-trace-java#8445) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🐛 Fix Scalatest tracing for tests that are reported asynchronously ([#8444](DataDog/dd-trace-java#8444) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Implement attempt to fix tests ([#8393](DataDog/dd-trace-java#8393) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Implement test disabling ([#8377](DataDog/dd-trace-java#8377) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Update CODEOWNERS parser to not log errors on comments with leading whitespace ([#8349](DataDog/dd-trace-java#8349) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Request Test Management tests list ([#8345](DataDog/dd-trace-java#8345) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Receive test management settings from CIVis settings request ([#8331](DataDog/dd-trace-java#8331) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Implement quarantined tests tagging ([#8326](DataDog/dd-trace-java#8326) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Implement tests quarantining ([#8320](DataDog/dd-trace-java#8320) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Add tag to specify if the user is setting DD_SERVICE ([#8318](DataDog/dd-trace-java#8318) - [@daniel-mohedano](https://github.com/daniel-mohedano)) ##### Crash tracking - ✨ Only fork jps when required ([#8419](DataDog/dd-trace-java#8419) - [@mcculls](https://github.com/mcculls)) - 🐛 Use Java home of the crashed process to launch crash uploader ([#8348](DataDog/dd-trace-java#8348) - [@jbachorik](https://github.com/jbachorik)) ##### Data Streams Monitoring - 🐛 Fix error happening when sqs message attributes are readonly ([#8473](DataDog/dd-trace-java#8473) - [@vandonr](https://github.com/vandonr)) - 🐛 Fix bug on proto schema extraction ([#8403](DataDog/dd-trace-java#8403) - [@vandonr](https://github.com/vandonr)) - 🐛 Fix service name overrides in consumers ([#8387](DataDog/dd-trace-java#8387) - [@piochelepiotr](https://github.com/piochelepiotr)) ##### Database Monitoring - ✨ Add DBMTracePreparedStatements to tracer configuration log ([#8508](DataDog/dd-trace-java#8508) - [@cecile75](https://github.com/cecile75)) ##### Dynamic Instrumentation - ✨ Look in another location for grpc service methods ([#8468](DataDog/dd-trace-java#8468) - [@evanchooly](https://github.com/evanchooly)) - 🐛 Fix Exception Replay with Lambda proxy classes ([#8452](DataDog/dd-trace-java#8452) - [@jpbempel](https://github.com/jpbempel)) - ✨ Add code origin support for spring-webmvc ([#8416](DataDog/dd-trace-java#8416) - [@evanchooly](https://github.com/evanchooly)) - ✨ Add support for scanning jar from loaded class ([#8370](DataDog/dd-trace-java#8370) - [@jpbempel](https://github.com/jpbempel)) - 🐛 Disable capture of entry values ([#8369](DataDog/dd-trace-java#8369) - [@jpbempel](https://github.com/jpbempel)) - 🐛 Fix CodeOrigin for `@Trace` annotation ([#8344](DataDog/dd-trace-java#8344) - [@jpbempel](https://github.com/jpbempel)) - 🐛 Fix equals/hashCode for CodeOrigin probe ([#8319](DataDog/dd-trace-java#8319) - [@jpbempel](https://github.com/jpbempel)) - ✨ Add code origin support to kafka message listeners ([#8301](DataDog/dd-trace-java#8301) - [@evanchooly](https://github.com/evanchooly)) ##### Metrics - ✨ Create metric: appsec.waf.error ([#8381](DataDog/dd-trace-java#8381) - [@sezen-datadog](https://github.com/sezen-datadog)) - ✨ Create metric: appsec.rasp.error ([#8364](DataDog/dd-trace-java#8364) - [@sezen-datadog](https://github.com/sezen-datadog)) ##### Profiling - ✨ Bump ddprof library to 1.22.0 ([#8463](DataDog/dd-trace-java#8463) - [@jbachorik](https://github.com/jbachorik)) - IBM J9 8u361 corresponds to OpenJDK 8u362 by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#187 - Fix compatibility with musl libc 1.2.4 by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#189 - Modify version extraction by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#179 - Do not write null values to jvminfo event by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#184 - Productize VMStructs-based stack walker by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#177 - A few minor downport issues by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#180 - Enable ASGCT by default on fairly safe J9 JDK versions by [@jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#181 - 🐛 Exclude OrderedThreadPoolExecutor from queue-time measurements ([#8456](DataDog/dd-trace-java#8456) - [@jbachorik](https://github.com/jbachorik)) - ✨ Record JVM info on JVMs without JFR ([#8431](DataDog/dd-trace-java#8431) - [@jbachorik](https://github.com/jbachorik)) - 🐛 Actually use CleanupTask in TempLocationManager ([#8420](DataDog/dd-trace-java#8420) - [@mcculls](https://github.com/mcculls)) - ✨ Only fork jps when required ([#8419](DataDog/dd-trace-java#8419) - [@mcculls](https://github.com/mcculls)) - 🐛 Adjust JFR checks for J9 ([#8405](DataDog/dd-trace-java#8405) - [@jbachorik](https://github.com/jbachorik)) - 🧹 Disable smap RSS parsing by default ([#8342](DataDog/dd-trace-java#8342) - [@MattAlp](https://github.com/MattAlp)) ##### Telemetry - 🐛 Add support for JBoss jar:file format to DependencyResolver ([#8428](DataDog/dd-trace-java#8428) - [@jandro996](https://github.com/jandro996)) - ✨ Update metrics: appsec.waf.requests ([#8353](DataDog/dd-trace-java#8353) - [@Mariovido](https://github.com/Mariovido)) ##### Trace context propagation - ✨ Introduce tracing propagator ([#8313](DataDog/dd-trace-java#8313) - [@PerfectSlayer](https://github.com/PerfectSlayer)) ##### Tracer core - 🐛 Fix Stable Config telemetry source names ([#8460](DataDog/dd-trace-java#8460) - [@BaptisteFoy](https://github.com/BaptisteFoy)) - ✨ Probe trace endpoints with a valid payload of empty arrays ([#8414](DataDog/dd-trace-java#8414) - [@mcculls](https://github.com/mcculls)) - ✨ Add 1 minute fail-safe to JUL/JMX class-loading callback ([#8399](DataDog/dd-trace-java#8399) - [@mcculls](https://github.com/mcculls)) - ✨ Migrate DSM injection calls to context-first APIs ([#8383](DataDog/dd-trace-java#8383) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - 🧹 Move continuation capture methods from scope to tracer ([#8371](DataDog/dd-trace-java#8371) - [@mcculls](https://github.com/mcculls)) - ✨ Migrate context extraction calls to context-first APIs ([#8368](DataDog/dd-trace-java#8368) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - 🧹 Migrate context injection calls to context-first APIs ([#8358](DataDog/dd-trace-java#8358) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - 💡 Support reading configurations from files ([#8338](DataDog/dd-trace-java#8338) - [@mtoffl01](https://github.com/mtoffl01)) - 💡 Implementation of BaggagePropagator and BaggageContext ([#8330](DataDog/dd-trace-java#8330) - [@mhlidd](https://github.com/mhlidd)) - 🧹 Combine continuation implementations into one which supports multiple activations ([#8324](DataDog/dd-trace-java#8324) - [@mcculls](https://github.com/mcculls)) - ✨ Introduce tracing propagator ([#8313](DataDog/dd-trace-java#8313) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - ✨ Remove old context propagation API ([#8271](DataDog/dd-trace-java#8271) - [@PerfectSlayer](https://github.com/PerfectSlayer)) ##### Instrumentations ##### AWS Lambda instrumentation - 🐛 Send error message and stack to Lambda extension ([#8417](DataDog/dd-trace-java#8417) - [@nhulston](https://github.com/nhulston)) ##### AWS SDK instrumentation - 🐛 Fix error happening when sqs message attributes are readonly ([#8473](DataDog/dd-trace-java#8473) - [@vandonr](https://github.com/vandonr)) - 💡 Inject trace context into AWS Step Functions input ([#7585](DataDog/dd-trace-java#7585) - [@DylanLovesCoffee](https://github.com/DylanLovesCoffee)) ##### Core Java language instrumentation - ✨ Look in another location for grpc service methods ([#8468](DataDog/dd-trace-java#8468) - [@evanchooly](https://github.com/evanchooly)) - ✨ Add code origin support for spring-webmvc ([#8416](DataDog/dd-trace-java#8416) - [@evanchooly](https://github.com/evanchooly)) - 💡 Implementation of BaggagePropagator and BaggageContext ([#8330](DataDog/dd-trace-java#8330) - [@mhlidd](https://github.com/mhlidd)) - ✨ Add code origin support to kafka message listeners ([#8301](DataDog/dd-trace-java#8301) - [@evanchooly](https://github.com/evanchooly)) ##### gRPC instrumentation - ✨ Look in another location for grpc service methods ([#8468](DataDog/dd-trace-java#8468) - [@evanchooly](https://github.com/evanchooly)) ##### Kafka instrumentation - ✨ Add messaging.destination.name tag to kafka integrations ([#8366](DataDog/dd-trace-java#8366) - [@rarguelloF](https://github.com/rarguelloF)) ##### Protocol Buffer instrumentation - 🐛 Fix bug on proto schema extraction ([#8403](DataDog/dd-trace-java#8403) - [@vandonr](https://github.com/vandonr)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - "after 6pm every weekday,before 2am every weekday" in timezone Australia/Melbourne, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Never, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). GitOrigin-RevId: 108a0f86aa59ab4c938cbac0688dd4c19cb301fa

Implement tests quarantining

af973a0

nikita-tkachenko-datadog added type: enhancement comp: ci visibility Continuous Integration Visibility labels Jan 31, 2025

Fix TestNG7 tests

6c76dc4

nikita-tkachenko-datadog marked this pull request as ready for review January 31, 2025 13:48

nikita-tkachenko-datadog requested a review from a team as a code owner January 31, 2025 13:48

nikita-tkachenko-datadog requested review from juan-fernandez and daniel-mohedano January 31, 2025 13:48

daniel-mohedano approved these changes Jan 31, 2025

View reviewed changes

nikita-tkachenko-datadog merged commit 64dca18 into master Jan 31, 2025
197 checks passed

nikita-tkachenko-datadog deleted the nikita-tkachenko/test-quarantine branch January 31, 2025 14:04

github-actions bot added this to the 1.47.0 milestone Jan 31, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement tests quarantining #8320

Implement tests quarantining #8320

nikita-tkachenko-datadog commented Jan 31, 2025 •

edited by jira bot

Loading

pr-commenter bot commented Jan 31, 2025 •

edited

Loading

Implement tests quarantining #8320

Implement tests quarantining #8320

Conversation

nikita-tkachenko-datadog commented Jan 31, 2025 • edited by jira bot Loading

What Does This Do

Contributor Checklist

pr-commenter bot commented Jan 31, 2025 • edited Loading

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

nikita-tkachenko-datadog commented Jan 31, 2025 •

edited by jira bot

Loading

pr-commenter bot commented Jan 31, 2025 •

edited

Loading