Querytee: Add metric to measure relative backend latency

[jhesketh]

This metric gives us a measurement of how individual queries compare between two backends in terms of latency (or duration).

What this PR does

Which issue(s) this PR fixes or relates to

Fixes #

Checklist

• Tests updated.
• Documentation added.
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
• about-versioning.md updated with experimental features.

[charleskorn]

Thanks for working on this.

It'd be good to add the proportional latency difference as well - a request being 1s slower is interesting, but knowing whether that's a 1% change or a 100% change would be useful as well.

[pracucci]

This metric gives us a measurement of how individual queries compare between two backends in terms of latency (or duration).

Why do we need a new metric? Can't we just solve this "problem" at PromQL level, with a query?

[jhesketh]

Why do we need a new metric? Can't we just solve this "problem" at PromQL level, with a query?

We currently measure the backend_request_duration_seconds as a histogram with a label for the backend. We can subtract one backend from the other to get a general indication of which one is more performant, but it isn't necessarily accurate as results can cancel each other out.

By tracking with a new metric, we can see on this individual histogram the number of times one backend is faster than the other (and by how much) depending if the bucket is positive or negative.

[charleskorn]

Overall LGTM, modulo comment on test changes.

Could you please add a changelog entry?

[jdbaldry]

Thanks for updating the metric name.

I think consistency with other metric descriptions is probably preferable to more clarity.

[charleskorn]

[nit] This method is only ever used with one value for name - might be able to simplify this and the test that calls it.