Hopefully fix intermittent CI failures

[heinrich5991]

Thanks to @Robyt3 for the solution.

https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/timeout.20investigation serde-rs/serde#2577
dtolnay/rust-toolchain#94

Checklist

• Tested the change ingame
• Provided screenshots if it is a visual change
• Tested in combination with possibly related configuration options
• Written a unit test (especially base/) or added coverage to integration test
• Considered possible null pointers and out of bounds array indexing
• Changed no physics that affect existing maps
• Tested the change with ASan+UBSan or valgrind's memcheck (optional)