From feb8ebcb104670c3091fc682b659d1f541135ec3 Mon Sep 17 00:00:00 2001 From: Adam Connelly Date: Tue, 27 Aug 2019 16:01:51 +0100 Subject: [PATCH] Update documentation for multi-resource changes (#686) - I've updated the documentation and the guide on adding a new scraper to match the changes made to support multiple resources per metric. - I also tweaked the wording about adding docs in the new scraper guide so it doesn't sound optional. I figured if someone goes to the bother of adding a scraper they can easily add some docs based on the examples that are there. - Tweaked some of the wording where I spotted a few typos / mistakes while proof-reading. - I've also tweaked the example metric names used. Some of them didn't make sense now that multiple resources are supported, and I figured it made sense to just do a pass and make them all consistent. Part of #513. --- adding-a-new-scraper.md | 27 ++++++------ .../v1.x/metrics/container-instances.md | 6 ++- .../v1.x/metrics/container-registry.md | 6 ++- docs/configuration/v1.x/metrics/cosmos-db.md | 18 ++++---- .../v1.x/metrics/generic-azure-resource.md | 13 +++--- docs/configuration/v1.x/metrics/index.md | 43 ++++++++----------- .../v1.x/metrics/network-interface.md | 18 ++++---- docs/configuration/v1.x/metrics/postgresql.md | 6 ++- .../configuration/v1.x/metrics/redis-cache.md | 6 ++- .../v1.x/metrics/service-bus-queue.md | 11 +++-- .../v1.x/metrics/storage-queue.md | 17 +++++--- .../v1.x/metrics/virtual-machine.md | 8 ++-- 12 files changed, 100 insertions(+), 79 deletions(-) diff --git a/adding-a-new-scraper.md b/adding-a-new-scraper.md index 27cb20cf3..630bab76c 100644 --- a/adding-a-new-scraper.md +++ b/adding-a-new-scraper.md @@ -8,25 +8,25 @@ This guide walks you through the process of adding a new scraper type. ------------------------- ## Configuration -1. Add your new scraping type to the `Promitor.Core.Scraping.Configuration.Model.ResourceType` -2. Describe the resource for which you're creating scraping metrics by creating `MetricDefinition`and inherit from `Promitor.Core.Scraping.Configuration.Model.Metrics.MetricDefinition` - this class should go in `.\src\Promitor.Core.Scraping\Configuration\Model\Metrics\ResourceTypes` -3. Create a new Deserializer in `.\src\Promitor.Core.Scraping\Configuration\Serialization\Deserializers`. This must inherit from `GenericAzureMetricDeserializer`. -4. Update `Promitor.Core.Scraping.Factories.MetricDeserializerFactory` to handle your new resource type by returning a new instance of the Deserializer you created in the previous step. -5. Provide a unit test in `.\src\Promitor.Scraper.Tests.Unit\Serialization\MetricsDeclaration\` that tests the deserialization based on our sample +1. Add your new scraping type to the `Promitor.Core.Scraping.Configuration.Model.ResourceType` +2. Describe the resource for which you're scraping metrics by creating `ResourceDefinition`and inherit from `Promitor.Core.Scraping.Configuration.Model.Metrics.AzureResourceDefinition` - this class should go in `.\src\Promitor.Core.Scraping\Configuration\Model\Metrics\ResourceTypes` +3. Create a new Deserializer in `.\src\Promitor.Core.Scraping\Configuration\Serialization\v1\Providers`. This must inherit from `ResourceDeserializer`. +4. Update `Promitor.Core.Scraping.Configuration.v1.Core.AzureResourceDeserializerFactory` to handle your new resource type by returning a new instance of the Deserializer you created in the previous step. +5. Provide a unit test in `.\src\Promitor.Scraper.Tests.Unit\Serialization\v1\Providers` that tests the deserialization based on our sample. Your test class must inherit from `ResourceDeserializerTest` to ensure the inherited functionality is tested. -Going forward in this guide, `TMetricDefinition` will refer to your newly created configuration type. +Going forward in this guide, `TResourceDefinition` will refer to your newly created configuration type. ## Validation For every scraper type we provide validation for the configuration so that Promitor fails to start up. This requires the following steps: -1. Create a new validator that implements `MetricValidator` where `TMetricDefinition` is the definition you made in step #2 above. This validator should reside in `.\src\Promitor.Scraper.Host\Validation\MetricDefinitions\ResourceTypes`. You can look at the contents of `ServiceBusQueueMetricValidator` for an idea of the validation inputs, steps, and outputs typical of validator implementation. +1. Create a new validator that implements `IMetricValidator`. This validator should reside in `.\src\Promitor.Scraper.Host\Validation\MetricDefinitions\ResourceTypes`. You can look at the contents of `ServiceBusQueueMetricValidator` for an idea of the validation inputs, steps, and outputs typical of validator implementation. 2. Add construction and usage of this validator to `.\src\Promitor.Scraper.Host\Validation\Factories\MetricValidatorFactory.cs` for the ResourceType you created in step #1 above. 3. Provide a unit test for every validation rule that was added in `.\src\Promitor.Scraper.Tests.Unit\Validation\Metrics\ResourceTypes` ## Scraping -1. Implement a scraper that inherits from `Scraper`. This one will specify how to call Azure Monitor. -2. Hook your new scraper in our `MetricScraperFactory` which determines what scraper to use for the passed configuration +1. Implement a scraper that inherits from `Scraper`. This will specify how to call Azure Monitor. +2. Hook your new scraper in our `MetricScraperFactory` which determines what scraper to use for the passed configuration. --------------------------- @@ -37,9 +37,10 @@ This requires the following steps: ## Documentation Features are great to have but without clear & well-written documentation they are somewhat useless. -It would be good if you could provide documentation on the following: -1. What Azure service it supports and how to use it -2. What fields need to be configured and what they are for -3. An example configuration +Please provide documentation on the following: + +1. What Azure service it supports and how to use it. +2. What fields need to be configured and what they are for. +3. An example configuration. This should be provided in a new file under `docs\configuration\metrics`. diff --git a/docs/configuration/v1.x/metrics/container-instances.md b/docs/configuration/v1.x/metrics/container-instances.md index 6a2c3196a..27e96cde9 100644 --- a/docs/configuration/v1.x/metrics/container-instances.md +++ b/docs/configuration/v1.x/metrics/container-instances.md @@ -14,14 +14,16 @@ All supported metrics are documented in the official [Azure Monitor documentatio Example: ```yaml -name: demo_containerinstances_cpu +name: azure_container_instance_cpu_usage description: "Average cpu usage of our 'promitor-container-instance' container instance" resourceType: ContainerInstance -containerGroup: promitor-container-instance azureMetricConfiguration: metricName: CpuUsage aggregation: type: Average +resources: +- containerGroup: promitor-container-instance-1 +- containerGroup: promitor-container-instance-2 ``` [← back to metrics declarations](/configuration/metrics)
diff --git a/docs/configuration/v1.x/metrics/container-registry.md b/docs/configuration/v1.x/metrics/container-registry.md index d1d37faaf..32720bc13 100644 --- a/docs/configuration/v1.x/metrics/container-registry.md +++ b/docs/configuration/v1.x/metrics/container-registry.md @@ -13,14 +13,16 @@ All supported metrics are not documented in the official [Azure Monitor document Example: ```yaml -name: demo_container_registry_total_pull_count +name: azure_container_registry_total_pull_count description: "Amount of images that were pulled from the container registry" resourceType: ContainerRegistry -registryName: promitor azureMetricConfiguration: metricName: TotalPullCount aggregation: type: Average +resources: +- registryName: promitor-1 +- registryName: promitor-2 ``` [← back to metrics declarations](/configuration/metrics)
diff --git a/docs/configuration/v1.x/metrics/cosmos-db.md b/docs/configuration/v1.x/metrics/cosmos-db.md index 6bcf56abf..3157fd8cc 100644 --- a/docs/configuration/v1.x/metrics/cosmos-db.md +++ b/docs/configuration/v1.x/metrics/cosmos-db.md @@ -13,14 +13,16 @@ All supported metrics are documented in the official [Azure Monitor documentatio Example: ```yaml - name: demo_cosmos_totalrequests - description: "Demo cosmos query" - resourceType: CosmosDb - dbName: cognitiveanalytics - azureMetricConfiguration: - metricName: TotalRequests - aggregation: - type: Count +name: azure_cosmos_db_total_requests +description: "Demo cosmos query" +resourceType: CosmosDb +azureMetricConfiguration: + metricName: TotalRequests + aggregation: + type: Count +resources: +- dbName: cognitiveanalytics-1 +- dbName: cognitiveanalytics-2 ``` [← back to metrics declarations](/configuration/metrics) diff --git a/docs/configuration/v1.x/metrics/generic-azure-resource.md b/docs/configuration/v1.x/metrics/generic-azure-resource.md index d4dd3bc03..bbefec0a7 100644 --- a/docs/configuration/v1.x/metrics/generic-azure-resource.md +++ b/docs/configuration/v1.x/metrics/generic-azure-resource.md @@ -14,17 +14,20 @@ Promitor simplifies defining resource uris by using the subscription & resource Example: ```yaml -name: demo_generic_queue_size +name: azure_service_bus_active_messages description: "Amount of active messages of the 'myqueue' queue (determined with Generic provider)" resourceType: Generic -# Will scrape subscriptions//resourceGroups//providers/Microsoft.ServiceBus/namespaces/promitor-messaging -# Where 'sub' & 'rg' are coming from azureMetadata -resourceUri: Microsoft.ServiceBus/namespaces/promitor-messaging -filter: EntityName eq 'orders' azureMetricConfiguration: metricName: ActiveMessages aggregation: type: Total +resources: +# Will scrape subscriptions//resourceGroups//providers/Microsoft.ServiceBus/namespaces/promitor-messaging +# Where 'sub' & 'rg' are coming from azureMetadata +- resourceUri: Microsoft.ServiceBus/namespaces/promitor-messaging + filter: EntityName eq 'orders' +- resourceUri: Microsoft.ServiceBus/namespaces/promitor-messaging + filter: EntityName eq 'items' ``` [← back to metrics declarations](/configuration/metrics)
diff --git a/docs/configuration/v1.x/metrics/index.md b/docs/configuration/v1.x/metrics/index.md index e3f5b4467..53a09dc81 100644 --- a/docs/configuration/v1.x/metrics/index.md +++ b/docs/configuration/v1.x/metrics/index.md @@ -3,7 +3,7 @@ layout: default title: Metrics Declaration --- -All the Azure Monitor metrics that needs to be scraped are consolidated in one YAML file. +All the Azure Monitor metrics that need to be scraped are consolidated in one YAML file. This configuration defines the Azure metadata and all the metrics. # General Declaration @@ -20,24 +20,26 @@ As Promitor evolves we need to change the structure of our metrics declaration. ## Metric Defaults -- `metricDefaults.scraping.schedule` - **[REQUIRED]** A cron expression that controls the fequency in which all the configured metrics will be scraped from Azure Monitor. You can use [crontab-generator.org](https://crontab-generator.org/) to generate a cron that fits your needs. +- `metricDefaults.scraping.schedule` - **[REQUIRED]** A cron expression that controls the frequency of which all the configured metrics will be scraped from Azure Monitor. You can use [crontab-generator.org](https://crontab-generator.org/) to generate a cron that fits your needs. - `metricDefaults.aggregation.interval` - The default interval which defines over what period measurements of a metric should be aggregated. ## Metrics Every metric that is being declared needs to define the following fields: -- `name` - Name of the metric that will be exposed in the scrape endpoint for Prometheus -- `description` - Description for the metric that will be exposed in the scrape endpoint for Prometheus +- `name` - Name of the metric that will be exposed in the scrape endpoint for Prometheus. +- `description` - Description for the metric that will be exposed in the scrape endpoint for Prometheus. - `resourceType` - Defines what type of resource needs to be queried. -- `labels` - Defines a set of custom labels to included for a given metric. +- `labels` - Defines a set of custom labels to include for a given metric. - `azureMetricConfiguration.metricName` - The name of the metric in Azure Monitor to query - `azureMetricConfiguration.aggregation.type` - The aggregation that needs to be used when querying Azure Monitor - `azureMetricConfiguration.aggregation.interval` - Overrides the default aggregation interval defined in `metricDefaults.aggregation.interval` with a new interval +- `resources` - An array of one or more resources to get metrics for. The fields required vary + depending on the `resourceType` being created, and are documented for each resource. All resources + support an optional `resourceGroupName` to allow the global resource group to be overridden. Additionally, the following fields are optional: -- `resourceGroupName` - The resource group to scrape for this metric. This allows you to specify a different resource group from the one configured in `azureMetadata`, enabling you to scrape multiple resource groups with one single configuration. - `scraping.schedule` - A scraping schedule for the individual metric; overrides the the one specified in `metricDefaults` ## Example @@ -56,11 +58,12 @@ metricDefaults: # Every minute schedule: "0 * * ? * *" metrics: - - name: demo_queue_size - description: "Amount of active messages of the 'myqueue' queue" + - name: azure_service_bus_active_messages + description: "The number of active messages on a service bus queue." resourceType: ServiceBusQueue - namespace: promitor-messaging - queueName: orders + labels: + app: promitor + tier: messaging scraping: # Every 2 minutes schedule: "0 */2 * ? * *" @@ -69,20 +72,12 @@ metrics: aggregation: type: Total interval: 00:15:00 - - name: demo_queue_dev_size - description: "Amount of active messages of the 'myqueue-dev' queue" - resourceType: ServiceBusQueue - namespace: promitor-messaging-dev - queueName: orders - resourceGroupName: promitor-dev - labels: - app: promitor - stage: dev - azureMetricConfiguration: - metricName: ActiveMessages - aggregation: - type: Total - interval: 00:15:00 + resources: + - namespace: promitor-messaging + queueName: orders + - namespace: promitor-messaging-dev + queueName: orders + resourceGroupName: promitor-dev ``` # Supported Azure Services diff --git a/docs/configuration/v1.x/metrics/network-interface.md b/docs/configuration/v1.x/metrics/network-interface.md index 80c0b11bc..89366c201 100644 --- a/docs/configuration/v1.x/metrics/network-interface.md +++ b/docs/configuration/v1.x/metrics/network-interface.md @@ -13,14 +13,16 @@ All supported metrics are documented in the official [Azure Monitor documentatio Example: ```yaml - - name: demo_azuresnetworkinterface_bytesreceivedrate - description: "Number of bytes the Network Interface sent" - resourceType: NetworkInterface - networkInterfaceName: promitor-network-interface - azureMetricConfiguration: - metricName: BytesReceivedRate - aggregation: - type: Average +name: azure_network_interface_bytes_received_rate +description: "Number of bytes the Network Interface sent" +resourceType: NetworkInterface +azureMetricConfiguration: + metricName: BytesReceivedRate + aggregation: + type: Average +resources: +- networkInterfaceName: promitor-network-interface-1 +- networkInterfaceName: promitor-network-interface-2 ``` [← back to metrics declarations](/configuration/metrics)
diff --git a/docs/configuration/v1.x/metrics/postgresql.md b/docs/configuration/v1.x/metrics/postgresql.md index 508404e68..a97e4679f 100644 --- a/docs/configuration/v1.x/metrics/postgresql.md +++ b/docs/configuration/v1.x/metrics/postgresql.md @@ -13,10 +13,9 @@ All supported metrics are documented in the official [Azure Monitor documentatio Example: ```yaml -name: postgre_sql_cpu_percent +name: azure_postgre_sql_cpu_percent description: "The CPU percentage on the server" resourceType: PostgreSql -serverName: Promitor scraping: schedule: "0 */2 * ? * *" azureMetricConfiguration: @@ -24,6 +23,9 @@ azureMetricConfiguration: aggregation: type: Average interval: 00:01:00 +resources: +- serverName: Promitor-1 +- serverName: Promitor-2 ``` [← back to metrics declarations](/configuration/metrics)
diff --git a/docs/configuration/v1.x/metrics/redis-cache.md b/docs/configuration/v1.x/metrics/redis-cache.md index c52bce95a..34c90093f 100644 --- a/docs/configuration/v1.x/metrics/redis-cache.md +++ b/docs/configuration/v1.x/metrics/redis-cache.md @@ -15,10 +15,9 @@ You can find more documentation on each metric in the Azure Cache for Redis [mon Example: ```yaml -name: redis_cache_hits +name: azure_redis_cache_cache_hits description: "The number of successful key lookups during the specified reporting interval. This maps to keyspace_hits from the Redis INFO command." resourceType: RedisCache -cacheName: Promitor scraping: schedule: "0 */2 * ? * *" azureMetricConfiguration: @@ -26,6 +25,9 @@ azureMetricConfiguration: aggregation: type: Total interval: 00:01:00 +resources: +- cacheName: Promitor-1 +- cacheName: Promitor-2 ``` [← back to metrics declarations](/configuration/metrics)
diff --git a/docs/configuration/v1.x/metrics/service-bus-queue.md b/docs/configuration/v1.x/metrics/service-bus-queue.md index b57e91e00..8b88667b4 100644 --- a/docs/configuration/v1.x/metrics/service-bus-queue.md +++ b/docs/configuration/v1.x/metrics/service-bus-queue.md @@ -17,15 +17,18 @@ The following scraper-specific metric label(s) will be added: Example: ```yaml -name: demo_queue_size -description: "Amount of active messages of the 'myqueue' queue" +name: azure_service_bus_queue_active_messages +description: "The number of active messages on a service bus queue" resourceType: ServiceBusQueue -namespace: promitor-messaging -queueName: orders azureMetricConfiguration: metricName: ActiveMessages aggregation: type: Total +resources: +- namespace: promitor-messaging + queueName: orders +- namespace: promitor-messaging + queueName: items ``` [← back to metrics declarations](/configuration/metrics)
diff --git a/docs/configuration/v1.x/metrics/storage-queue.md b/docs/configuration/v1.x/metrics/storage-queue.md index 97fe32d1e..35e26b968 100644 --- a/docs/configuration/v1.x/metrics/storage-queue.md +++ b/docs/configuration/v1.x/metrics/storage-queue.md @@ -22,17 +22,22 @@ The following scraper-specific metric label(s) will be added: Example: ```yaml -name: demo_queue_size -description: "Amount of messages in the 'orders' queue" +name: azure_storage_queue_message_count +description: "The number of messages on an Azure storage queue" resourceType: StorageQueue -accountName: promitor -queueName: orders -sasToken: - environmentVariable: "SECRETS_STORAGEQUEUE_PROMITOR_SASTOKEN" azureMetricConfiguration: metricName: MessageCount aggregation: type: Total +resources: +- accountName: promitor + queueName: orders + sasToken: + environmentVariable: "SECRETS_STORAGEQUEUE_PROMITOR_SASTOKEN" +- accountName: promitor + queueName: items + sasToken: + environmentVariable: "SECRETS_STORAGEQUEUE_PROMITOR_SASTOKEN" ``` [← back to metrics declarations](/configuration/metrics)
diff --git a/docs/configuration/v1.x/metrics/virtual-machine.md b/docs/configuration/v1.x/metrics/virtual-machine.md index 519f497c9..3d964d119 100644 --- a/docs/configuration/v1.x/metrics/virtual-machine.md +++ b/docs/configuration/v1.x/metrics/virtual-machine.md @@ -13,14 +13,16 @@ All supported metrics are documented in the official [Azure Monitor documentatio Example: ```yaml -name: demo_virtualmachine_percentage_cpu -description: "Average percentage cpu usage of our 'promitor-virtual-machine' virtual machine" +name: azure_virtual_machine_percentage_cpu +description: "Average percentage cpu usage on an Azure virtual machine" resourceType: VirtualMachine -virtualMachineName: promitor-virtual-machine azureMetricConfiguration: metricName: Percentage CPU aggregation: type: Average +resources: +- virtualMachineName: promitor-virtual-machine-1 +- virtualMachineName: promitor-virtual-machine-2 ``` [← back to metrics declarations](/configuration/metrics)