Upgrade Rust toolchain to nightly-2024-03-01
#3052
Conversation
celinval
changed the title
nightly-2024-01-03nightly-2024-03-01
|
Please just make sure there's no significant performance degradation before merging. |
|
Unfortunately I'm getting consistent >400% regressions in |
|
can you manually run before and after the PR to see if that's the case? |
|
This test is known to have high variance in performance, e.g. see https://github.com/model-checking/kani/actions/runs/7453782015 The numbers in this PR are on par with the numbers in previous PRs, so I would consider this to be noise. You can confirm by running the test locally a number of times with and without the changes. |
adpaco-aws
force-pushed
the
toolchain-upgrade-2024-01-03
branch
from
24419a0
to
ca58557
Compare
|
Thanks all for your feedback. I've run each version 10 times and the results are:
So it looks like there's a performance regression of about +50s/+33% for this test. |
|
After some discussion, we consider that the regression is due to SAT variability. No data suggests that the problem is more difficult per se after the toolchain upgrade. For example, runtime symex remains constant in all cases (~0.8s), as does program size (16292 steps) and VCCs (615/231). The table below shows SAT-solving time for the UNSATISFIABLE instance in each of the iterations.
If we compare this with the table posted in my previous comment, we can conclude that the total time increase is due to the increase in SAT-solving time for the UNSATISFIABLE instance. Therefore, there is no reason to block this toolchain upgrade. @zhassan-aws pointed out that there's almost no difference before/after this toolchain upgrade if we use CaDiCaL instead (the current solver is Kissat). So we'll propose switching the solver to CaDiCaL for this test. |
These are the original release notes for the reference: ## What's Changed * Automatic cargo update to 2024-02-26 by @github-actions in #3043 * Upgrade rust toolchain to 2024-02-17 by @celinval in #3040 * Upgrade `windows-targets` crate to version 0.52.4 by @adpaco-aws in #3049 * Fix `codegen_atomic_binop` for `atomic_ptr` by @qinheping in #3047 * Upgrade Rust toolchain to `nightly-2024-02-25` by @adpaco-aws in #3048 * Update s2n-quic submodule by @zhassan-aws in #3050 * Update s2n-quic submodule weekly through dependabot by @zhassan-aws in #3053 * Retrieve info for recursion tracker reliably by @feliperodri in #3045 * Automatic cargo update to 2024-03-04 by @github-actions in #3055 * Upgrade Rust toolchain to `nightly-2024-03-01` by @adpaco-aws in #3052 * Add `--use-local-toolchain` to Kani setup by @jaisnan in #3056 * Replace internal reverse_postorder by a stable one by @celinval in #3064 * Add option to override `--crate-name` from `kani` by @adpaco-aws in #3054 * cargo update and fix macos CI by @zhassan-aws in #3067 * Bump tests/perf/s2n-quic from `d103836` to `1a7faa8` by @dependabot in #3066 * Upgrade toolchain to 2024-03-11 by @zhassan-aws in #3071 * Emit `dead` goto-instructions on MIR StatementDead by @karkhaz in #3063 **Full Changelog**: kani-0.47.0...kani-0.48.0
Upgrades the Rust toolchain to
nightly-2024-03-01. The Rust compiler PRs that triggered changes in this upgrades are:By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.