Is it possible to compare perf on minor revisions of the .net runtime or SDK?

[marklam]

I can compare my current .net9 vs .net8 easily enough, but is there a way I can compare runtime 8.0.11 (8.0.1124.51707) vs 8.0.12 (8.0.1224.60305)?

Can I similarly compare performance of the code when compiled with SDK 9.0.100 vs 9.0.102 ?

[adamsitnik]

Hello @marklam

Yes, it's possible. You can specify a path to the dotnet.exe by using the --cli command line argument.

BTW the .NET Team is tracking performance with BenchmarkDotNet in https://github.com/dotnet/performance repository to ensure that there are no regressions. If there is any scenario that is not covered with the benchmarks please open an issue and share the details.

Thanks,
Adam

[marklam]

@adamsitnik - Am I right that this would require two runs of the benchmark for A vs B, rather than getting reports on different jobs within the same dotnet run as you would with Jobs?

I'll be sure to report any regression if I can prove it exists :-)

[timcassell]

You can pass multiple jobs to the config with toolchains using CsProjCoreToolchain.From(NetCoreAppSettings)

[marklam]

Well, there's no perf regression. Just me making a dumb error.

But for anybody in the future with the same question (including future me), you need to download and unzip the "Binaries" packages for the SDKs you want to test, and then set up the benchmark run like:

    let net8011 = NetCoreAppSettings.NetCoreApp80.WithCustomDotNetCliPath(@"C:\Users\markl\Downloads\dotnet-sdk-8.0.111-win-x64\dotnet.exe")
    let net8012 = NetCoreAppSettings.NetCoreApp80.WithCustomDotNetCliPath(@"C:\Users\markl\Downloads\dotnet-sdk-8.0.112-win-x64\dotnet.exe")

    let config =
        DefaultConfig.Instance
            .AddJob(Job.Default.WithToolchain(CsProjCoreToolchain.From(net8011)).WithId("net8.0.11")) //.NET 8.0.11 (8.0.1124.51707)
            .AddJob(Job.Default.WithToolchain(CsProjCoreToolchain.From(net8012)).WithId("net8.0.12")) //.NET 8.0.12 (8.0.1224.60305)

    let summary = BenchmarkRunner.Run<InvokationBenchmarks>(config)