Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump Timeit version to v0.1.14 #4766

Merged
merged 4 commits into from
Oct 26, 2023
Merged

Conversation

tonyredondo
Copy link
Member

@tonyredondo tonyredondo commented Oct 25, 2023

Summary of changes

This PR bumps the version of timeit tool from v0.1.3 to v0.1.14

Verified

This commit was signed with the committer’s verified signature.
tonyredondo Tony Redondo
@tonyredondo tonyredondo self-assigned this Oct 25, 2023
@tonyredondo tonyredondo requested a review from a team as a code owner October 25, 2023 19:47
@github-actions github-actions bot added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Oct 25, 2023

Verified

This commit was signed with the committer’s verified signature.
tonyredondo Tony Redondo
@tonyredondo tonyredondo changed the title Bump Timeit version to v0.1.12 Bump Timeit version to v0.1.13 Oct 25, 2023
fix

Verified

This commit was signed with the committer’s verified signature.
tonyredondo Tony Redondo
@tonyredondo tonyredondo changed the title Bump Timeit version to v0.1.13 Bump Timeit version to v0.1.14 Oct 25, 2023

Verified

This commit was signed with the committer’s verified signature.
tonyredondo Tony Redondo
@DataDog DataDog deleted a comment from datadog-ddstaging bot Oct 26, 2023
@DataDog DataDog deleted a comment from andrewlock Oct 26, 2023
@DataDog DataDog deleted a comment from andrewlock Oct 26, 2023
@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Oct 26, 2023

Datadog Report

Branch report: tony/bump-timeitsharp-to-v0.1.12
Commit report: aad8af4

dd-trace-dotnet: 0 Failed, 0 New Flaky, 304153 Passed, 941 Skipped, 30m 20.38s Wall Time

@andrewlock
Copy link
Member

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

Loading
gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4766) - mean (68ms)  : 66, 71
     .   : milestone, 68,

    section CallTarget+Inlining+NGEN
    This PR (4766) - mean (981ms)  : 961, 1000
     .   : milestone, 981,

Loading
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4766) - mean (106ms)  : 104, 108
     .   : milestone, 106,

    section CallTarget+Inlining+NGEN
    This PR (4766) - mean (670ms)  : 655, 685
     .   : milestone, 670,

Loading
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4766) - mean (90ms)  : 89, 92
     .   : milestone, 90,

    section CallTarget+Inlining+NGEN
    This PR (4766) - mean (633ms)  : 619, 647
     .   : milestone, 633,

Loading
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4766) - mean (187ms)  : 185, 190
     .   : milestone, 187,
    master - mean (189ms)  : 187, 191
     .   : milestone, 189,

    section CallTarget+Inlining+NGEN
    This PR (4766) - mean (1,124ms)  : 1101, 1148
     .   : milestone, 1124,
    master - mean (1,135ms)  : 1117, 1153
     .   : milestone, 1135,

Loading
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4766) - mean (271ms)  : 266, 276
     .   : milestone, 271,
    master - mean (273ms)  : 270, 277
     .   : milestone, 273,

    section CallTarget+Inlining+NGEN
    This PR (4766) - mean (1,088ms)  : 1070, 1106
     .   : milestone, 1088,
    master - mean (1,094ms)  : 1074, 1115
     .   : milestone, 1094,

Loading
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4766) - mean (261ms)  : 257, 265
     .   : milestone, 261,
    master - mean (264ms)  : 259, 268
     .   : milestone, 264,

    section CallTarget+Inlining+NGEN
    This PR (4766) - mean (1,055ms)  : 1031, 1079
     .   : milestone, 1055,
    master - mean (1,064ms)  : 1041, 1087
     .   : milestone, 1064,

@andrewlock
Copy link
Member

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

Loading
gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4766) (11.708M)   : 0, 11707990
    master (10.849M)   : 0, 10848650
    benchmarks/2.38.0 (11.867M)   : 0, 11866956
    benchmarks/2.9.0 (11.404M)   : 0, 11403964

    section Automatic
    This PR (4766) (8.225M)   : 0, 8225187
    master (7.689M)   : 0, 7688765
    benchmarks/2.38.0 (8.176M)   : 0, 8175636
    benchmarks/2.9.0 (8.122M)   : 0, 8121934

    section Trace stats
    This PR (4766) (8.419M)   : 0, 8418804
    master (7.974M)   : 0, 7973986
    benchmarks/2.38.0 (8.450M)   : 0, 8450370

    section Manual
    This PR (4766) (10.397M)   : 0, 10397073
    master (9.776M)   : 0, 9775925
    benchmarks/2.38.0 (10.334M)   : 0, 10334368

    section Manual + Automatic
    This PR (4766) (7.721M)   : 0, 7720994
    master (7.300M)   : 0, 7299779
    benchmarks/2.38.0 (7.750M)   : 0, 7750484

    section Version Conflict
    This PR (4766) (7.035M)   : 0, 7034738
    master (6.575M)   : 0, 6574617
    benchmarks/2.38.0 (7.137M)   : 0, 7136691

Loading
gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4766) (9.537M)   : 0, 9537150
    master (9.534M)   : 0, 9534375
    benchmarks/2.38.0 (9.548M)   : 0, 9548121
    benchmarks/2.9.0 (9.656M)   : 0, 9655806

    section Automatic
    This PR (4766) (6.798M)   : 0, 6798424
    master (6.740M)   : 0, 6739842
    benchmarks/2.38.0 (6.747M)   : 0, 6747227

    section Trace stats
    This PR (4766) (6.952M)   : 0, 6951643
    master (6.775M)   : 0, 6774905
    benchmarks/2.38.0 (6.815M)   : 0, 6814846

    section Manual
    This PR (4766) (8.438M)   : 0, 8438164
    master (8.302M)   : 0, 8302488
    benchmarks/2.38.0 (8.263M)   : 0, 8263131

    section Manual + Automatic
    This PR (4766) (6.116M)   : 0, 6115847
    master (6.349M)   : 0, 6348627
    benchmarks/2.38.0 (6.275M)   : 0, 6275411

    section Version Conflict
    This PR (4766) (5.480M)   : 0, 5480441
    master (5.749M)   : 0, 5749334
    benchmarks/2.38.0 (5.670M)   : 0, 5669744

Loading
gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4766) (7.112M)   : 0, 7111617
    master (9.450M)   : 0, 9449674
    benchmarks/2.38.0 (9.007M)   : 0, 9007066
    benchmarks/2.9.0 (9.731M)   : 0, 9730905

    section Automatic
    This PR (4766) (6.602M)   : 0, 6602341
    master (6.635M)   : 0, 6634884
    benchmarks/2.38.0 (6.423M)   : 0, 6422756
    benchmarks/2.9.0 (6.908M)   : 0, 6908188

    section Trace stats
    This PR (4766) (6.844M)   : 0, 6844275
    master (6.864M)   : 0, 6863822
    benchmarks/2.38.0 (6.721M)   : 0, 6720838

    section Manual
    This PR (4766) (8.229M)   : 0, 8228654
    master (8.250M)   : 0, 8250092
    benchmarks/2.38.0 (7.977M)   : 0, 7977494

    section Manual + Automatic
    This PR (4766) (6.512M)   : 0, 6511883
    master (6.462M)   : 0, 6462400
    benchmarks/2.38.0 (6.349M)   : 0, 6349066

    section Version Conflict
    This PR (4766) (5.830M)   : 0, 5830396
    master (5.794M)   : 0, 5794375
    benchmarks/2.38.0 (5.682M)   : 0, 5681654

Loading
gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    master (7.751M)   : 0, 7750914
    benchmarks/2.38.0 (7.578M)   : 0, 7577544
    benchmarks/2.9.0 (7.904M)   : 0, 7903721

    section No attack
    master (2.190M)   : 0, 2190399
    benchmarks/2.38.0 (2.184M)   : 0, 2184279
    benchmarks/2.9.0 (3.239M)   : 0, 3238804

    section Attack
    master (1.724M)   : 0, 1723833
    benchmarks/2.38.0 (1.711M)   : 0, 1710542
    benchmarks/2.9.0 (2.459M)   : 0, 2458912

    section Blocking
    master (3.465M)   : 0, 3464934
    benchmarks/2.38.0 (3.486M)   : 0, 3486124

    section IAST default
    master (6.848M)   : 0, 6847970

    section IAST full
    master (6.223M)   : 0, 6222736

    section Base vuln
    master (0.962M)   : 0, 961814

    section IAST vuln
    master (0.904M)   : 0, 903897

@tonyredondo tonyredondo merged commit c98c4a8 into master Oct 26, 2023
@tonyredondo tonyredondo deleted the tony/bump-timeitsharp-to-v0.1.12 branch October 26, 2023 14:21
@github-actions github-actions bot added this to the vNext milestone Oct 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:builds project files, build scripts, pipelines, versioning, releases, packages
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants