Provide support for scraping multiple resource groups

[datadot]

Currently the metrics configuration is as follows:

azureMetadata:
  tenantId: c8819874-9e56-4e3f-b1a8-1c0325138f27
  subscriptionId: 0f9d7fea-99e8-4768-8672-06a28514f77e
  resourceGroupName: promitor
metricDefaults:
  aggregation:
    interval: 00:05:00
metrics:
  - name: demo_generic_queue_size
    description: "Amount of active messages of the 'myqueue' queue (determined with Generic provider)"
    resourceType: Generic
    resourceUri: Microsoft.ServiceBus/namespaces/promitor-messaging
    filter: EntityName eq 'orders'
    azureMetricConfiguration:
      metricName: ActiveMessages
      aggregation:
        type: Total

This works for the scenario of monitoring a single set of metrics within one resource group, however best practices dictates that not all resources are located within a single resource group within a subscription. Moving the resourceGroupName to the metrics configuration would provide support for metrics from multiple resource groups within Azure.

Example configuration:

azureMetadata:
  tenantId: c8819874-9e56-4e3f-b1a8-1c0325138f27
  subscriptionId: 0f9d7fea-99e8-4768-8672-06a28514f77e
metricDefaults:
  aggregation:
    interval: 00:05:00
metrics:
  - name: demo_generic_queue_size_rg1
    description: "Amount of active messages of the 'myqueue' queue (determined with Generic provider)"
    resourceType: Generic
    resourceGroupName: RG_1
    resourceUri: Microsoft.ServiceBus/namespaces/promitor-messaging
    filter: EntityName eq 'orders'
    azureMetricConfiguration:
      metricName: ActiveMessages
      aggregation:
        type: Total
  - name: demo_generic_queue_size_rg2
    description: "Amount of active messages of the 'myqueue' queue (determined with Generic provider)"
    resourceType: Generic
    resourceGroupName: RG_2
    resourceUri: Microsoft.ServiceBus/namespaces/promitor-messaging
    filter: EntityName eq 'orders'
    azureMetricConfiguration:
      metricName: ActiveMessages
      aggregation:
        type: Total

[tomkerkhove]

That's a fair ask, but I have limited bandwidth so won't be able to deliver it quickly.

I agree on your suggestion, but would still keep it in the azureMetadata as a default value where you can override it per metric. The reason is that I would like to avoid duplication.

Does that make sense to you?

[tomkerkhove]

As a workaround you can deploy an instance per resource group and configure different scraping paths per RG - https://promitor.io/configuration/#scraping

[datadot]

Agreed with work around and suggested enhancement approach.

Thanks

[tomkerkhove]

Actually I was giving this another thought and it feels like the current scraping boundaries are good in terms of one instance is one resource group. What is your concern about running multiple instances or is it just because of convenience? Imagine that you could scrape from multiple RGs, who will deploy Promitor and build the config? Wouldn't it make more sense to deploy an instance along with your app and only limited to that RG? Building on top of that, will the services be part of same kubernetes namespace? Just interested in the scenario here.

[datadot]

So my current pipelines are:

Dev
QA
Performance
Stage
Prod 1....x

My plan is a 2 Prometheus instances, 1 for the Dev\QA\Performance\Stage running preferably off single Promitor instance. I dynamically build the metrics yaml within the pipelines, so was hoping to have a single Promitor instance for the development environments. We will also be running a separate management AKS cluster for monitoring and management activities away from the live clusters which will monitor all production queues again spread across multiple resource groups.

I personally wouldn't agree that using Resource Groups are a suitable boundary and technically you can use them for multiple different containment reasons within Azure.

Building on top of that, will the services be part of same Kubernetes namespace?

Within our deployment the Kubernetes namespace is ultimately irrelevant and doesn't align to any back end azure resource group.

[tomkerkhove]

Well it's best practice to scope them to your unit of deployment but that is not important here, you run your platform how you want!

I'm going to provide this, but it will only be as part of v1.0 which can take a little longer.

It will probably look like:

azureMetadata:
  tenantId: c8819874-9e56-4e3f-b1a8-1c0325138f27
  subscriptionId: 0f9d7fea-99e8-4768-8672-06a28514f77e
  resourceGroup: RG_1
aggregation:
  interval: 00:05:00
metrics:
  - name: demo_generic_queue_size_rg1
    description: "Amount of active messages of the 'myqueue' queue (determined with Generic provider)"
    resourceType: Generic
    resourceUri: Microsoft.ServiceBus/namespaces/promitor-messaging
    filter: EntityName eq 'orders'
    azureMetricConfiguration:
      metricName: ActiveMessages
      aggregation:
        type: Total
  - name: demo_generic_queue_size_rg2
    description: "Amount of active messages of the 'myqueue' queue (determined with Generic provider)"
    resourceType: Generic
    resourceGroupName: RG_2
    resourceUri: Microsoft.ServiceBus/namespaces/promitor-messaging
    filter: EntityName eq 'orders'
    azureMetricConfiguration:
      metricName: ActiveMessages
      aggregation:
        type: Total

You can define a general resource group but still override it on the metric itself.

However, this brings me to subscriptions. Are you using the same subscription or not?

[tomkerkhove]

Last chance for feedback @datadot

[datadot]

Override is good idea, keeps existing configuration support.
I'd say subscription can be a separate instance, I think that's a fair expectation.

[tomkerkhove]

Great to hear, thanks for the feedback @datadot! I'll make the changes in next coming week(s)

[brusMX]

dibs