Report seed for failed Proptest tests in CI

[shekhirin]

Describe the feature

If proptest fails, we have no insight into how to reproduce the failure. We should report the seed used for the test run.

I'm sure there's a native way to do so in proptest-rs.

Additional context

No response

[greged93]

hey @shekhirin I'm down to work on this

[greged93]

looking into proptest a little, it seems it will by default reports the seed in stdout and create a proptest-regressions with this failing seed.

It is then recommended to commit this file in order to avoid regression. We could have the CI open a PR on failing proptests to add the failing seed to the concerned proptest-regresssions file but might be a little much. Let me know where you want to go with this @shekhirin.

• https://proptest-rs.github.io/proptest/proptest/failure-persistence.html

EDIT:
stdout example of failing proptest:

---- tests::test_addition stdout ----
thread 'tests::test_addition' panicked at src/lib.rs:9:13:
This is a panic
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
proptest: Saving this and future failures in redacted/random/proptest-regressions/lib.txt
proptest: If this test was run on a CI system, you may wish to add the following line to your copy of the file. (You may need to create it.)
cc c07c212ae0bd6647204636668dc32ff126dcc474a3bcafbf9e0e42ae55e3afc8
thread 'tests::test_addition' panicked at src/lib.rs:5:5:
Test failed: This is a panic.
minimal failing input: a = 0, b = 0
        successes: 0
        local rejects: 0
        global rejects: 0

[shekhirin]

@greged93 here's a failure that we recently had https://github.com/paradigmxyz/reth/actions/runs/10615954550/job/29425196986

Now that I look deeper into it, I think it's not a proptest failure, but just a benchmark failing due to bad test vectors that were generated with proptest

reth/crates/cli/commands/src/test_vectors/tables.rs

Lines 82 to 83 in e1a7012

	let strategy =
	proptest::collection::vec(arb::<TableRow<T>>(), per_table - rows.len()).no_shrink().boxed();

What we should do is to print to stdout the seed that was used to generate test vectors, so we can reproduce the error in benchmarks locally.

[greged93]

ah got it, will look into getting the seed out of benchmarks