Implement prometheus-style tags filters

[melan]

Prometheus supports several very useful operations to match time series by sets of labels: https://prometheus.io/docs/prometheus/latest/querying/operators/. It would be great if Argus can support the same type of operations.

Here are few examples when this can be useful:

Let's say we have memory usage by applications in different datacenters:

mem.used{dc=dc1, host=host1-1, app=my-app-1}
mem.used{dc=dc1, host=host1-1, app=my-app-2}
mem.used{dc=dc1, host=host1-2, app=my-app-1}
mem.used{dc=dc1, host=host1-2, app=my-app-2}
mem.used{dc=dc2, host=host2-1, app=my-app-1}
mem.used{dc=dc2, host=host2-1, app=my-app-2}
mem.used{dc=dc3, host=host3-1, app=my-app-1}
mem.used{dc=dc3, host=host3-1, app=my-app-2}

mem.total{dc=dc1, host=host1-1, host_type=small}
mem.total{dc=dc1, host=host1-2, host_type=small}
mem.total{dc=dc2, host=host2-1, host_type=medium}
mem.total{dc=dc3, host=host3-1, host_type=small}

If I want to get an average of memory utilization by dc for dc1 and dc2. I want to use a construction like:

avg(sum(mem.used{dc=~"dc1|dc2"}) by (dc, host) / on(dc, host) mem.total{dc=~"dc1|dc2"}) by (dc) as avg_mem_percent_used

I would expect this expression to produce 2 time series: avg_mem_percent_used{dc=dc1} and avg_mem_percent_used{dc=dc2}

What does it do:

• mem.used{dc=~"dc1|dc2"} - selects all mem.used metrics for dc tag is either dc1 or dc2
• sum(mem.used{dc=~"dc1|dc2"}) by (dc, host) - computes total of mem.used by dc and host tags. Something like SELECT sum(mem.used) WHERE dc in ('dc1', 'dc2') GROUP BY dc, host
• sum(mem.used{dc=~"dc1|dc2"}) by (dc, host) / on(dc, host) mem.total{dc=~"dc1|dc2"} - match result time series from the last step and mem.total by tags with values. on(dc, host) tells to match by dc and host tags only. mem.total has an extra tag host_type so it'll be ignored.
• avg(sum(mem.used{dc=~"dc1|dc2"}) by (dc, host) / on(dc, host) mem.total{dc=~"dc1|dc2"}) by (dc) - will take results from the previous step, will group them by dc tag and will compute average for every every group
• ... as avg_mem_percent_used - will name result metrics as avg_mem_percent_used, tags will be taken from the previous step, it will be dc tag.

This syntax makes expressions much easier to read and understand.

[melan]

/cc: @reecemarkowsky