Ingester: Limit read path based on CPU/memory utilization

[aknuds1]

What this PR does

In the ingester, limit read path based on CPU/memory utilization. Read requests are rejected if the configured CPU/memory utilization threshold is exceeded, with HTTP code 503 (service unavailable).

CPU and memory utilization are obtained from one of the following, in order:

1. cgroup v2
2. cgroup v1
3. /proc/self

The implementation is Linux only, for simplicity. The feature is marked as experimental though, so it should be OK to not have a cross-platform solution (yet?).

CPU utilization is calculated as an exponential weighted moving average over the last minute.

TODO

• Should we measure CPU and memory utilization on the process or system level? I'm currently uncertain of whether system CPU/memory utilization within a container tells you the container utilization or that of the host
• Are there more direct ways of finding CPU and memory utilization under Docker or k8s?
• Figure out why gateway seems to translate 503 to 500
• Add tests

Which issue(s) this PR fixes or relates to

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[jhalterman]

Could we just fetch current CPU usage via the library?

[aknuds1]

Interesting, hadn't seen that. In any case, we likely won't be using gopsutil by the time the PR is finalized. I'll try to learn from the Percent implementation.

[aknuds1]

Also, the Percent function isn't suitable, since it calculates the system's CPU utilization and not that of a single process (we want to know how much CPU the ingester process is using).

[aknuds1]

I see on the other hand that there's a Process.Percent method. I'll investigate its implementation.

[aknuds1]

Having checked the Process.Percent implementation, it's essentially the same as mine AFAICT. Though, I wonder why the cpu.Percent function subtracts Guest and GuestNice from the total CPU time, while process.Process.Percent does not?? Guess it warrants some research.

[jhalterman]

Yea, cpu.Percent might only be good to use if Mimir is the only container running in an ingester pod, which it seems to be, at least in the Helm chart.

[aknuds1]

Yeah, it's generally alone in its pod. I've never experienced otherwise. Also, Mimir can run outside of Docker/k8s, so have to support that usage (as opposed to using some Docker/k8s method of finding CPU load).

[aknuds1]

(Note that since the original comment was made, the PR has changed into measuring utilization via cgroup v2 and cgroup v1 before it measures via the process' resource utilization.)