Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support delayed trigger of local triage report #6479

Merged
merged 2 commits into from
Jan 16, 2024
Merged

Support delayed trigger of local triage report #6479

merged 2 commits into from
Jan 16, 2024

Conversation

mcculls
Copy link
Contributor

@mcculls mcculls commented Jan 12, 2024 •
edited by jira bot
Loading

To schedule a triage report two minutes after startup add this JVM option:

-Ddd.triage.report.trigger=2m

or set this environment variable:

DD_TRIAGE_REPORT_TRIGGER=2m

Units of h (hours) and s (seconds) are also accepted, the default unit when omitted is seconds.

The report will be written to $TMPDIR by default, to choose a different location use:

-Ddd.triage.report.dir=/example/reports/

or

DD_TRIAGE_REPORT_DIR=/example/reports/

the report zip will be named dd-java-flare-<epoch-time-in-millis>.zip

Motivation

For situations where remote-config or the tracer-flare endpoint is not available but we'd like to capture a triage report.

Jira ticket: APMJAVA-1078

@mcculls mcculls added the tag: diagnostics Diagnostics related changes label Jan 12, 2024
@pr-commenter
Copy link

pr-commenter bot commented Jan 12, 2024 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master mcculls/trigger-local-triage-report
git_commit_date 1705412999 1705413279
git_commit_sha b6069c5 50519c7
release_version 1.28.0-SNAPSHOT~b6069c585f 1.28.0-SNAPSHOT~50519c7b33
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1705416129 1705416129
ci_job_id 409398067 409398067
ci_pipeline_id 26524646 26524646
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 45 metrics, 9 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.28.0-SNAPSHOT~50519c7b33, baseline=1.28.0-SNAPSHOT~b6069c585f

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.054 s) : 0, 1054485
Total [baseline] (9.383 s) : 0, 9382695
Agent [candidate] (1.054 s) : 0, 1053862
Total [candidate] (9.38 s) : 0, 9380224
section appsec
Agent [baseline] (1.158 s) : 0, 1158128
Total [baseline] (9.47 s) : 0, 9469778
Agent [candidate] (1.156 s) : 0, 1156208
Total [candidate] (9.538 s) : 0, 9538486
section iast
Agent [baseline] (1.176 s) : 0, 1175694
Total [baseline] (9.551 s) : 0, 9550552
Agent [candidate] (1.179 s) : 0, 1178598
Total [candidate] (9.587 s) : 0, 9586780
section profiling
Agent [baseline] (1.291 s) : 0, 1291208
Total [baseline] (9.687 s) : 0, 9686871
Agent [candidate] (1.282 s) : 0, 1281957
Total [candidate] (9.608 s) : 0, 9608370
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.054 s -
Agent appsec 1.158 s 103.642 ms (9.8%)
Agent iast 1.176 s 121.209 ms (11.5%)
Agent profiling 1.291 s 236.723 ms (22.4%)
Total tracing 9.383 s -
Total appsec 9.47 s 87.083 ms (0.9%)
Total iast 9.551 s 167.858 ms (1.8%)
Total profiling 9.687 s 304.176 ms (3.2%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.054 s -
Agent appsec 1.156 s 102.346 ms (9.7%)
Agent iast 1.179 s 124.736 ms (11.8%)
Agent profiling 1.282 s 228.096 ms (21.6%)
Total tracing 9.38 s -
Total appsec 9.538 s 158.262 ms (1.7%)
Total iast 9.587 s 206.556 ms (2.2%)
Total profiling 9.608 s 228.146 ms (2.4%) 
gantt
    title petclinic - break down per module: candidate=1.28.0-SNAPSHOT~50519c7b33, baseline=1.28.0-SNAPSHOT~b6069c585f

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (665.629 ms) : 0, 665629
BytebuddyAgent [candidate] (665.878 ms) : 0, 665878
GlobalTracer [baseline] (296.107 ms) : 0, 296107
GlobalTracer [candidate] (295.309 ms) : 0, 295309
AppSec [baseline] (50.644 ms) : 0, 50644
AppSec [candidate] (50.546 ms) : 0, 50546
Remote Config [baseline] (663.121 µs) : 0, 663
Remote Config [candidate] (653.394 µs) : 0, 653
Telemetry [baseline] (7.209 ms) : 0, 7209
Telemetry [candidate] (7.241 ms) : 0, 7241
section appsec
BytebuddyAgent [baseline] (669.565 ms) : 0, 669565
BytebuddyAgent [candidate] (667.337 ms) : 0, 667337
GlobalTracer [baseline] (297.375 ms) : 0, 297375
GlobalTracer [candidate] (297.586 ms) : 0, 297586
AppSec [baseline] (149.185 ms) : 0, 149185
AppSec [candidate] (149.404 ms) : 0, 149404
Remote Config [baseline] (651.147 µs) : 0, 651
Remote Config [candidate] (651.937 µs) : 0, 652
Telemetry [baseline] (6.926 ms) : 0, 6926
Telemetry [candidate] (6.938 ms) : 0, 6938
section iast
BytebuddyAgent [baseline] (775.268 ms) : 0, 775268
BytebuddyAgent [candidate] (775.65 ms) : 0, 775650
GlobalTracer [baseline] (286.439 ms) : 0, 286439
GlobalTracer [candidate] (287.319 ms) : 0, 287319
AppSec [baseline] (51.962 ms) : 0, 51962
AppSec [candidate] (52.743 ms) : 0, 52743
Remote Config [baseline] (583.925 µs) : 0, 584
Remote Config [candidate] (594.876 µs) : 0, 595
Telemetry [baseline] (6.61 ms) : 0, 6610
Telemetry [candidate] (8.912 ms) : 0, 8912
IAST [baseline] (20.42 ms) : 0, 20420
IAST [candidate] (19.092 ms) : 0, 19092
section profiling
ProfilingAgent [baseline] (129.171 ms) : 0, 129171
ProfilingAgent [candidate] (126.072 ms) : 0, 126072
BytebuddyAgent [baseline] (668.696 ms) : 0, 668696
BytebuddyAgent [candidate] (664.841 ms) : 0, 664841
GlobalTracer [baseline] (378.935 ms) : 0, 378935
GlobalTracer [candidate] (377.522 ms) : 0, 377522
AppSec [baseline] (51.255 ms) : 0, 51255
AppSec [candidate] (50.87 ms) : 0, 50870
Remote Config [baseline] (990.032 µs) : 0, 990
Remote Config [candidate] (977.041 µs) : 0, 977
Telemetry [baseline] (7.341 ms) : 0, 7341
Telemetry [candidate] (7.287 ms) : 0, 7287
Profiling [baseline] (129.196 ms) : 0, 129196
Profiling [candidate] (126.096 ms) : 0, 126096
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.28.0-SNAPSHOT~50519c7b33, baseline=1.28.0-SNAPSHOT~b6069c585f

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.054 s) : 0, 1054252
Total [baseline] (8.732 s) : 0, 8732168
Agent [candidate] (1.056 s) : 0, 1056390
Total [candidate] (8.744 s) : 0, 8743751
section iast
Agent [baseline] (1.178 s) : 0, 1177954
Total [baseline] (9.263 s) : 0, 9263226
Agent [candidate] (1.184 s) : 0, 1184261
Total [candidate] (9.275 s) : 0, 9274655
section iast_TELEMETRY_OFF
Agent [baseline] (1.169 s) : 0, 1168922
Total [baseline] (9.27 s) : 0, 9270243
Agent [candidate] (1.17 s) : 0, 1170463
Total [candidate] (9.251 s) : 0, 9250740
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.054 s -
Agent iast 1.178 s 123.702 ms (11.7%)
Agent iast_TELEMETRY_OFF 1.169 s 114.67 ms (10.9%)
Total tracing 8.732 s -
Total iast 9.263 s 531.058 ms (6.1%)
Total iast_TELEMETRY_OFF 9.27 s 538.075 ms (6.2%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.056 s -
Agent iast 1.184 s 127.871 ms (12.1%)
Agent iast_TELEMETRY_OFF 1.17 s 114.073 ms (10.8%)
Total tracing 8.744 s -
Total iast 9.275 s 530.904 ms (6.1%)
Total iast_TELEMETRY_OFF 9.251 s 506.989 ms (5.8%) 
gantt
    title insecure-bank - break down per module: candidate=1.28.0-SNAPSHOT~50519c7b33, baseline=1.28.0-SNAPSHOT~b6069c585f

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (665.5 ms) : 0, 665500
BytebuddyAgent [candidate] (667.242 ms) : 0, 667242
GlobalTracer [baseline] (295.925 ms) : 0, 295925
GlobalTracer [candidate] (296.265 ms) : 0, 296265
AppSec [baseline] (50.63 ms) : 0, 50630
AppSec [candidate] (50.589 ms) : 0, 50589
Remote Config [baseline] (661.845 µs) : 0, 662
Remote Config [candidate] (666.39 µs) : 0, 666
Telemetry [baseline] (7.264 ms) : 0, 7264
Telemetry [candidate] (7.259 ms) : 0, 7259
section iast
BytebuddyAgent [baseline] (774.817 ms) : 0, 774817
BytebuddyAgent [candidate] (781.002 ms) : 0, 781002
GlobalTracer [baseline] (287.268 ms) : 0, 287268
GlobalTracer [candidate] (288.593 ms) : 0, 288593
AppSec [baseline] (53.758 ms) : 0, 53758
AppSec [candidate] (55.576 ms) : 0, 55576
Remote Config [baseline] (592.035 µs) : 0, 592
Remote Config [candidate] (583.986 µs) : 0, 584
Telemetry [baseline] (6.591 ms) : 0, 6591
Telemetry [candidate] (6.615 ms) : 0, 6615
IAST [baseline] (20.492 ms) : 0, 20492
IAST [candidate] (17.298 ms) : 0, 17298
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (767.638 ms) : 0, 767638
BytebuddyAgent [candidate] (769.263 ms) : 0, 769263
GlobalTracer [baseline] (286.502 ms) : 0, 286502
GlobalTracer [candidate] (286.408 ms) : 0, 286408
AppSec [baseline] (53.536 ms) : 0, 53536
AppSec [candidate] (54.948 ms) : 0, 54948
Remote Config [baseline] (575.374 µs) : 0, 575
Remote Config [candidate] (577.38 µs) : 0, 577
Telemetry [baseline] (6.404 ms) : 0, 6404
Telemetry [candidate] (6.367 ms) : 0, 6367
IAST [baseline] (20.005 ms) : 0, 20005
IAST [candidate] (18.533 ms) : 0, 18533
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-01-16T14:21:29 2024-01-16T14:37:56
git_branch master mcculls/trigger-local-triage-report
git_commit_date 1705412999 1705413279
git_commit_sha b6069c5 50519c7
release_version 1.28.0-SNAPSHOT~b6069c585f 1.28.0-SNAPSHOT~50519c7b33
start_time 2024-01-16T14:21:16 2024-01-16T14:37:43
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1705416129 1705416129
ci_job_id 409398067 409398067
ci_pipeline_id 26524646 26524646
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 8 metrics, 14 unstable metrics.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.28.0-SNAPSHOT~50519c7b33, baseline=1.28.0-SNAPSHOT~b6069c585f
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.343 ms) : 1324, 1363
.   : milestone, 1343,
appsec (1.765 ms) : 1739, 1790
.   : milestone, 1765,
iast (1.528 ms) : 1504, 1553
.   : milestone, 1528,
profiling (1.491 ms) : 1465, 1516
.   : milestone, 1491,
tracing (1.506 ms) : 1482, 1530
.   : milestone, 1506,
section candidate
no_agent (1.341 ms) : 1322, 1360
.   : milestone, 1341,
appsec (1.747 ms) : 1722, 1772
.   : milestone, 1747,
iast (1.517 ms) : 1492, 1542
.   : milestone, 1517,
profiling (1.518 ms) : 1492, 1543
.   : milestone, 1518,
tracing (1.5 ms) : 1475, 1524
.   : milestone, 1500,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.343 ms [1.324 ms, 1.363 ms] -
appsec 1.765 ms [1.739 ms, 1.79 ms] 421.046 µs (31.3%)
iast 1.528 ms [1.504 ms, 1.553 ms] 184.815 µs (13.8%)
profiling 1.491 ms [1.465 ms, 1.516 ms] 147.228 µs (11.0%)
tracing 1.506 ms [1.482 ms, 1.53 ms] 162.86 µs (12.1%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.341 ms [1.322 ms, 1.36 ms] -
appsec 1.747 ms [1.722 ms, 1.772 ms] 405.371 µs (30.2%)
iast 1.517 ms [1.492 ms, 1.542 ms] 175.801 µs (13.1%)
profiling 1.518 ms [1.492 ms, 1.543 ms] 176.346 µs (13.1%)
tracing 1.5 ms [1.475 ms, 1.524 ms] 158.426 µs (11.8%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.28.0-SNAPSHOT~50519c7b33, baseline=1.28.0-SNAPSHOT~b6069c585f
    dateFormat X
    axisFormat %s
section baseline
no_agent (368.001 µs) : 347, 389
.   : milestone, 368,
iast (473.674 µs) : 453, 494
.   : milestone, 474,
iast_FULL (534.6 µs) : 514, 555
.   : milestone, 535,
iast_INACTIVE (440.368 µs) : 420, 461
.   : milestone, 440,
iast_TELEMETRY_OFF (472.251 µs) : 451, 494
.   : milestone, 472,
tracing (435.185 µs) : 415, 456
.   : milestone, 435,
section candidate
no_agent (369.885 µs) : 350, 390
.   : milestone, 370,
iast (490.335 µs) : 470, 511
.   : milestone, 490,
iast_FULL (535.782 µs) : 515, 556
.   : milestone, 536,
iast_INACTIVE (461.756 µs) : 441, 482
.   : milestone, 462,
iast_TELEMETRY_OFF (474.249 µs) : 453, 495
.   : milestone, 474,
tracing (443.457 µs) : 422, 464
.   : milestone, 443,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 368.001 µs [346.86 µs, 389.141 µs] -
iast 473.674 µs [453.032 µs, 494.316 µs] 105.673 µs (28.7%)
iast_FULL 534.6 µs [514.224 µs, 554.977 µs] 166.6 µs (45.3%)
iast_INACTIVE 440.368 µs [420.164 µs, 460.573 µs] 72.368 µs (19.7%)
iast_TELEMETRY_OFF 472.251 µs [450.639 µs, 493.863 µs] 104.25 µs (28.3%)
tracing 435.185 µs [414.723 µs, 455.646 µs] 67.184 µs (18.3%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 369.885 µs [349.758 µs, 390.013 µs] -
iast 490.335 µs [469.679 µs, 510.991 µs] 120.45 µs (32.6%)
iast_FULL 535.782 µs [515.27 µs, 556.294 µs] 165.896 µs (44.9%)
iast_INACTIVE 461.756 µs [441.039 µs, 482.472 µs] 91.87 µs (24.8%)
iast_TELEMETRY_OFF 474.249 µs [453.474 µs, 495.025 µs] 104.364 µs (28.2%)
tracing 443.457 µs [422.444 µs, 464.471 µs] 73.572 µs (19.9%)

@mcculls mcculls force-pushed the mcculls/trigger-local-triage-report branch 3 times, most recently from 74317d0 to ea0d306 Compare January 12, 2024 15:24
@mcculls mcculls marked this pull request as ready for review January 12, 2024 16:00
@mcculls mcculls requested a review from a team as a code owner January 12, 2024 16:00
@mcculls mcculls requested review from ygree, am312, PerfectSlayer and nayeem-kamal and removed request for ygree January 12, 2024 16:00
@mcculls mcculls force-pushed the mcculls/trigger-local-triage-report branch from ea0d306 to d8d03c6 Compare January 15, 2024 08:55
@mcculls mcculls force-pushed the mcculls/trigger-local-triage-report branch 3 times, most recently from 0466e0a to ba47807 Compare January 16, 2024 10:34
@mcculls mcculls force-pushed the mcculls/trigger-local-triage-report branch from ba47807 to 50519c7 Compare January 16, 2024 13:54
@mcculls mcculls enabled auto-merge (squash) January 16, 2024 16:52
@mcculls mcculls merged commit f47b31f into master Jan 16, 2024
@mcculls mcculls deleted the mcculls/trigger-local-triage-report branch January 16, 2024 17:10
@github-actions github-actions bot added this to the 1.28.0 milestone Jan 16, 2024
@PerfectSlayer PerfectSlayer added the comp: core Tracer core label Jan 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@PerfectSlayer PerfectSlayer PerfectSlayer left review comments

@am312 am312 am312 approved these changes

@nayeem-kamal nayeem-kamal Awaiting requested review from nayeem-kamal

Assignees
No one assigned
Labels
comp: core Tracer core tag: diagnostics Diagnostics related changes
Projects
None yet
Milestone
1.28.0
Development

Successfully merging this pull request may close these issues.
3 participants
@mcculls @PerfectSlayer @am312