Optimize some CI stuff

[jackcmay]

Problem

• Rebuilds
• Not running integrations tests sequentially
• Slow overall
• Tasks uses cuda when they don't need it

Summary of Changes

• Re-ordered and re-grouped builds so that like builds built together (features together, etc...)
• Fixed how we run integration tests sequentially. -jobs was being used but that only affects how many threads are used to build the tests. We want many threads building but only one thread running a test at a time. Changed from -jobs to --test-threads. This means we build fast but run each test sequentially in a single thread.
• Not using beta 1.31 yet, it does cut down on the rebuilds but it also takes a lot longer to compile. Reported an issue to cargo: Cargo rebuilds workspace member when building root even if workspace member was just built rust-lang/cargo#6337
• Buildkite runs multiple tasks in parallel so the total time to run is the time it takes to run the longest task. Bench was taking ~30 mins in Docker. Not running in docker brought it down to ~15 mins which are about the same as the next longest task so overall CI times are down from ~30 mins to ~15 mins. Also, the single cargo bench command is taking pretty much the entire ~15 mins. Could split that out into multiple Buildkite tasks and do them in parallel.
• Schedule stable-perf first so cuda not grabbed and hogged by another build task that does not need it. Should result in more predictable CI times.
• Break out clippy into its own task to run in parallel since it takes 10 mins alone.

More details in the comments below...

Fixes #

[jackcmay]

Was able to knock 4 minutes off of overall CI by localizing the feature builds to stable-perf (which we should probably label stable-features or something).

But bench was still taking 26 minutes to build, all of it essentially here:
cargo bench --features=unstable --verbose -- -Z unstable-options --format=json | 26m 25s |

I tried it without docker and the times went down to 15 minutes! Since buildkite runs in parallel, the overall time it takes for CI to finish is the longest task. The 2nd running task is stable which takes 15 minutes which means that bringing down bench from 26 to 15 minutes resulted in bringing down the entire CI time by 10 minutes or by approximately a third.

We could probably save even more by either balancing out work amount the other buildkite tasks or creating additional buildkite tasks to parallelize the load.

The results are in the following build: https://buildkite.com/solana-labs/solana/builds/5239

[jackcmay]

Got more concrete times on beta, something is weird because I would not expect beta to take this much longer, maybe beta itself is not optimized, or built as release, or something.

• cargo build took 13+ minutes instead of 2
• cargo test took 2 minutes instead of 1

The results are in the following build: https://buildkite.com/solana-labs/solana/builds/5243

[jackcmay]

@mvines Let me know what you think