Add warmup period to CPU utilization moving average

[pracucci]

What this PR does

Yesterday I enabled the CPU-based limiting in a production cell and I noticed requests to be limited right after the ingester startup, even if the 1m CPU utilization average (as computed by rate(container_cpu_usage_seconds_total[1m])) was way below the limit (about the half).

After some investigation, I realised that the limit triggered right after the limiter was started. We start the limiter after we replay the TSDB WALs in the ingester, and right before the ingester is ready to accept samples. Looking at logs of ingesters where the limit triggered, they all show the same pattern:

ts=2023-06-30T18:07:17.632955811Z caller=ingester.go:2210 level=info msg="successfully opened existing TSDBs"
ts=2023-06-30T18:07:17.633186146Z caller=mimir.go:834 level=info msg="Application started"
ts=2023-06-30T18:07:19.633923839Z caller=utilization.go:149 level=info context="read path" msg="enabling resource utilization based limiting" reason=cpu memory_limit=21474836480 memory_utilization=5136072704 cpu_limit=8 cpu_utilization=8.61
ts=2023-06-30T18:07:22.634063682Z caller=utilization.go:153 level=info context="read path" msg="disabling resource utilization based limiting" memory_limit=21474836480 memory_utilization=5138870272 cpu_limit=8 cpu_utilization=7.892821998317039

While reading https://github.com/VividCortex/ewma#readme to learn more how the EWMA works, I reliased that to get more stable results you should have a warmup period (which is also implemented by VividCortex's VariableEWMA). The idea of the warmup period is that for the first N samples, you add samples to the EWMA but you don't consider the computed rate as valid.

In this PR I'm adding a warmup period to our UtilizationBasedLimiter. I've also added a test to show the CPU utilization "sensitivity" (I just assert on the max CPU utilization because that's what we care with regards to the limiting).

Which issue(s) this PR fixes or relates to

N/A

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[jhalterman]

This fix is well spotted and looks good, but it might be better to add the warmup support directly into EwmaRate, so that it doesn't need to be implemented externally and future users don't hit the same problem.

[aknuds1]

Good catch! Please see my suggested minor simplification.

Also, @jhalterman's suggestion is worth considering.

[pracucci]

Thanks for the review. Addressed @aknuds1 suggestions. I think @jhalterman suggestion is a good idea, but will keep it for a follow up PR, given we're currently in a hurry to get this PR merged.

[grafanabot]

The backport to r245 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-5394-to-r245 origin/r245
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x f86040d0b31d89de4ed225adbaf24853a2b05c80
# Push it to GitHub
git push --set-upstream origin backport-5394-to-r245
git switch main
# Remove the local backport branch
git branch -D backport-5394-to-r245

Then, create a pull request where the base branch is r245 and the compare/head branch is backport-5394-to-r245.