[DX] There isn’t a single local command to ensure all CI will pass

[nagisa]

We identified flaky tests and similar such issues as a pain point back at the Washington protocol team meetup. We have made significant improvements on that front and now have significantly fewer flaky tests, and yet the developer experience still seems to be in a pretty terrible spot, and the remaining flaky tests are just a part of a problem.

In particular I find myself having a significant back-and-forth with the CI every time I make a pull request. Sometimes, yes, it is just a flaky test that requires a re-run, but more often than not it is a game of a whack-a-mole with all different test configurations that the CI runs. If I make sure that cargo nextest works locally, then the CI is more likely than not to complain about some integration test or some test that only runs with --features=nightly, if I run my local tests with --features=nightly and enable integration tests, then the regular non-nightly tests will explode. Even things as simple as formatting are a pain, as I find myself waiting 15 minutes just to find out I forgot to run cargo fmt for the last formatting touch-ups.

This last one I made an attempt to improve upon in #9290, but the general observation I have is that, unless I set up my development workflow to effectively replicate what buildkite does, I will be fighting the CI all the time. And I don’t want to do any of that. I want to cargo nextest, git push and move on with my day.

I would argue that if everybody ran the entire test suite all the time locally, we’d also somewhat naturally figure out the issues to flaky tests and #9367 as well.

[nagisa]

It would be great if we could find some resources to improve on this. We could also have a (very) short session to probe other engineers if they're happy/unhappy about their developer experience and where their pain points are if any.

cc @akhi3030 @gmilescu @walnut-the-cat

[nikurt]

To ensure my PR will pass the sanity check I'm running this

cargo fmt && cargo check --workspace --all-targets --all-features && ./scripts/run_clippy.sh && cargo build -pneard

Integration tests nightly/not-nightly, and nayduck are separate, and that is not convenient.

[nagisa]

just test-ci should give everyone a pretty good level of confidence nowadays, although it is pretty slow to run…