metrics: evaluate and reduce 'noise' in metrics tests

[grahamwhaley]

Some of our tests are quite 'noisy' in their results. That makes then tricky to use for CI.
Evaluate the noise in the tests, particularly the tests we use in the CI.
Evaluate (starting with the noisiest first maybe) where the noise comes from, and what, if anything, we can do to mitigate it.

Random notes:

• network and storage tests tend to be noisy - in some ways that is the nature of the beast for those.
• running the tests multiple times and aggregating reduces noise, but can take a substantial time
• there are a number of things we can contemplate to reduce noise, including
  • better use of the tests (investigate the available test program arguments for instance)
  • pinning to CPUs - inside and outside the container
  • pinning power save states on the host
  • smarter use of the test results (some tests generate quartiles etc., and we presently ignore them)