Provide support for multi-dimensional metrics

[tomkerkhove]

Provide support for scraping multi-dimensional metrics.

Configuration should be expanded to include optional dimension information.

This should cover:

• Name of the dimension to use (mandatory)
• Target of partition in dimension you are interested in (optional)

The reported metric in the scraping endpoint should use labels to report the metric with a general name but labeled for every dimension regardless if a target was specified or not.

Names are just examples, open for feedback!

Example

name: demo_containerinstances_cpu
description: "Average cpu usage of our 'promitor-container-instance' container instance"
resourceType: ContainerInstance
containerGroup: promitor-container-instance
azureMetricConfiguration:
  metricName: CpuUsage
  aggregation:
    type: Average
  dimension:
    name: containerName # name of the dimension to split on
    target: redis-single-node # name of the targetted partition in the dimension you're interested in

Portal

You can see this in action in the Azure Portal:

[tomkerkhove]

This will be picked up by @michellemcone

[melmaliacone]

@tomkerkhove
So I did some digging by playing around with things the in portal and it looks like applying splitting just sends a HTTP request with a filter of dimension_name eq '*' and returns the data for each item split. If you want a specific target, you would just apply a filter of dimension_name eq target_name. I think it's still worthwhile to include, but I would implement it using the same logic as filter and check if dimension_name and target_name are valid for the resource. I would also make target_name optional, so if it's not included it would return metrics for all of the targets, split by dimension_name.

[melmaliacone]

I finally found documentation that explains what I'm talking about here.

Essentially the portal experience is inconsistent with the way you would get multi-dimensional metrics via an HTTP request. Apply splitting is actually just a filter on a dimension that returns the list of metrics split on. If you want a specific target, it's exactly the same as add filter.

[melmaliacone]

Also, someone else can pick this up. I won't be working on it past this week.

[tomkerkhove]

@michellemcone Thanks for the thorough analysis and your feedback on it!

I would also make target_name optional, so if it's not included it would return metrics for all of the targets, split by dimension_name.

Wouldn't that mean we're actually not using a dimension then or how would that work?

Care to elaborate on that and/or include a portal example please?

[brusMX]

This is something we definitely need to implement for our customers. How should we proceed to support multi-dimension, is it just enough to add it as a filter expression and document it?

[tomkerkhove]

You can just add a filter when doing the Azure Monitor call. For example, when you add containerName eq 'sidecar' it will only show the metrics for container sidecar in a whole ACI container group.

Frankly, I'm not planning this for v1.0 but if you want to contribute it I can spec it out so that you can get started.

This impacts:

• Validation
• Scraping & labelling
• Docs

[tomkerkhove]

Actually, I was wrong as I was talking about filtering data per name, but splitting multi-dimensional metrics are similar.

When specifying the EntityName eq '*' filter it will give me all queues for a given namespace.

We should use this approach.

[tomkerkhove]

First rough version is working.

Without dimension:

# HELP promitor_demo_container_cpu The CPU of our containers in a container group.
# TYPE promitor_demo_container_cpu gauge
promitor_demo_container_cpu{resource_group="promitor-sources",subscription_id="0f9d7fea-99e8-4768-8672-06a28514f77e",resource_uri="subscriptions/0f9d7fea-99e8-4768-8672-06a28514f77e/resourceGroups/promitor-sources/providers/Microsoft.ContainerInstance/containerGroups/promitor-container-group",instance_name="promitor-container-group"} 0.5 1578063505227

With dimension on containerName:

# HELP promitor_demo_container_cpu The CPU of our containers in a container group.
# TYPE promitor_demo_container_cpu gauge
promitor_demo_container_cpu{resource_group="promitor-sources",subscription_id="0f9d7fea-99e8-4768-8672-06a28514f77e",resource_uri="subscriptions/0f9d7fea-99e8-4768-8672-06a28514f77e/resourceGroups/promitor-sources/providers/Microsoft.ContainerInstance/containerGroups/promitor-container-group",instance_name="promitor-container-group",containerName="aci-tutorial-app"} 0 1578063614577
promitor_demo_container_cpu{resource_group="promitor-sources",subscription_id="0f9d7fea-99e8-4768-8672-06a28514f77e",resource_uri="subscriptions/0f9d7fea-99e8-4768-8672-06a28514f77e/resourceGroups/promitor-sources/providers/Microsoft.ContainerInstance/containerGroups/promitor-container-group",instance_name="promitor-container-group",containerName="aci-tutorial-sidecar"} 1 1578063613819