Skip to content

Commit

Permalink
Update documentation for multi-resource changes (#686)
Browse files Browse the repository at this point in the history
- I've updated the documentation and the guide on adding a new scraper to match the changes made to support multiple resources per metric.
- I also tweaked the wording about adding docs in the new scraper guide so it doesn't sound optional. I figured if someone goes to the bother of adding a scraper they can easily add some docs based on the examples that are there.
- Tweaked some of the wording where I spotted a few typos / mistakes while proof-reading.
- I've also tweaked the example metric names used. Some of them didn't make sense now that multiple resources are supported, and I figured it made sense to just do a pass and make them all consistent.

Part of #513.
  • Loading branch information
adamconnelly authored and tomkerkhove committed Aug 27, 2019
1 parent fce24c8 commit feb8ebc
Show file tree
Hide file tree
Showing 12 changed files with 100 additions and 79 deletions.
27 changes: 14 additions & 13 deletions adding-a-new-scraper.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,25 +8,25 @@ This guide walks you through the process of adding a new scraper type.
-------------------------

## Configuration
1. Add your new scraping type to the `Promitor.Core.Scraping.Configuration.Model.ResourceType`
2. Describe the resource for which you're creating scraping metrics by creating `<New-Type>MetricDefinition`and inherit from `Promitor.Core.Scraping.Configuration.Model.Metrics.MetricDefinition` - this class should go in `.\src\Promitor.Core.Scraping\Configuration\Model\Metrics\ResourceTypes`
3. Create a new Deserializer in `.\src\Promitor.Core.Scraping\Configuration\Serialization\Deserializers`. This must inherit from `GenericAzureMetricDeserializer`.
4. Update `Promitor.Core.Scraping.Factories.MetricDeserializerFactory` to handle your new resource type by returning a new instance of the Deserializer you created in the previous step.
5. Provide a unit test in `.\src\Promitor.Scraper.Tests.Unit\Serialization\MetricsDeclaration\` that tests the deserialization based on our sample
1. Add your new scraping type to the `Promitor.Core.Scraping.Configuration.Model.ResourceType`
2. Describe the resource for which you're scraping metrics by creating `<New-Type>ResourceDefinition`and inherit from `Promitor.Core.Scraping.Configuration.Model.Metrics.AzureResourceDefinition` - this class should go in `.\src\Promitor.Core.Scraping\Configuration\Model\Metrics\ResourceTypes`
3. Create a new Deserializer in `.\src\Promitor.Core.Scraping\Configuration\Serialization\v1\Providers`. This must inherit from `ResourceDeserializer`.
4. Update `Promitor.Core.Scraping.Configuration.v1.Core.AzureResourceDeserializerFactory` to handle your new resource type by returning a new instance of the Deserializer you created in the previous step.
5. Provide a unit test in `.\src\Promitor.Scraper.Tests.Unit\Serialization\v1\Providers` that tests the deserialization based on our sample. Your test class must inherit from `ResourceDeserializerTest` to ensure the inherited functionality is tested.

Going forward in this guide, `TMetricDefinition` will refer to your newly created configuration type.
Going forward in this guide, `TResourceDefinition` will refer to your newly created configuration type.

## Validation
For every scraper type we provide validation for the configuration so that Promitor fails to start up.

This requires the following steps:
1. Create a new validator that implements `MetricValidator<TMetricDefinition>` where `TMetricDefinition` is the definition you made in step #2 above. This validator should reside in `.\src\Promitor.Scraper.Host\Validation\MetricDefinitions\ResourceTypes`. You can look at the contents of `ServiceBusQueueMetricValidator` for an idea of the validation inputs, steps, and outputs typical of validator implementation.
1. Create a new validator that implements `IMetricValidator`. This validator should reside in `.\src\Promitor.Scraper.Host\Validation\MetricDefinitions\ResourceTypes`. You can look at the contents of `ServiceBusQueueMetricValidator` for an idea of the validation inputs, steps, and outputs typical of validator implementation.
2. Add construction and usage of this validator to `.\src\Promitor.Scraper.Host\Validation\Factories\MetricValidatorFactory.cs` for the ResourceType you created in step #1 above.
3. Provide a unit test for every validation rule that was added in `.\src\Promitor.Scraper.Tests.Unit\Validation\Metrics\ResourceTypes`

## Scraping
1. Implement a scraper that inherits from `Scraper<TMetricDefinition>`. This one will specify how to call Azure Monitor.
2. Hook your new scraper in our `MetricScraperFactory` which determines what scraper to use for the passed configuration
1. Implement a scraper that inherits from `Scraper<TResourceDefinition>`. This will specify how to call Azure Monitor.
2. Hook your new scraper in our `MetricScraperFactory` which determines what scraper to use for the passed configuration.

---------------------------

Expand All @@ -37,9 +37,10 @@ This requires the following steps:
## Documentation
Features are great to have but without clear & well-written documentation they are somewhat useless.

It would be good if you could provide documentation on the following:
1. What Azure service it supports and how to use it
2. What fields need to be configured and what they are for
3. An example configuration
Please provide documentation on the following:

1. What Azure service it supports and how to use it.
2. What fields need to be configured and what they are for.
3. An example configuration.

This should be provided in a new file under `docs\configuration\metrics`.
6 changes: 4 additions & 2 deletions docs/configuration/v1.x/metrics/container-instances.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,16 @@ All supported metrics are documented in the official [Azure Monitor documentatio

Example:
```yaml
name: demo_containerinstances_cpu
name: azure_container_instance_cpu_usage
description: "Average cpu usage of our 'promitor-container-instance' container instance"
resourceType: ContainerInstance
containerGroup: promitor-container-instance
azureMetricConfiguration:
metricName: CpuUsage
aggregation:
type: Average
resources:
- containerGroup: promitor-container-instance-1
- containerGroup: promitor-container-instance-2
```
[&larr; back to metrics declarations](/configuration/metrics)<br />
Expand Down
6 changes: 4 additions & 2 deletions docs/configuration/v1.x/metrics/container-registry.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,16 @@ All supported metrics are not documented in the official [Azure Monitor document

Example:
```yaml
name: demo_container_registry_total_pull_count
name: azure_container_registry_total_pull_count
description: "Amount of images that were pulled from the container registry"
resourceType: ContainerRegistry
registryName: promitor
azureMetricConfiguration:
metricName: TotalPullCount
aggregation:
type: Average
resources:
- registryName: promitor-1
- registryName: promitor-2
```
[&larr; back to metrics declarations](/configuration/metrics)<br />
Expand Down
18 changes: 10 additions & 8 deletions docs/configuration/v1.x/metrics/cosmos-db.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,16 @@ All supported metrics are documented in the official [Azure Monitor documentatio

Example:
```yaml
name: demo_cosmos_totalrequests
description: "Demo cosmos query"
resourceType: CosmosDb
dbName: cognitiveanalytics
azureMetricConfiguration:
metricName: TotalRequests
aggregation:
type: Count
name: azure_cosmos_db_total_requests
description: "Demo cosmos query"
resourceType: CosmosDb
azureMetricConfiguration:
metricName: TotalRequests
aggregation:
type: Count
resources:
- dbName: cognitiveanalytics-1
- dbName: cognitiveanalytics-2
```
[&larr; back to metrics declarations](/configuration/metrics)
Expand Down
13 changes: 8 additions & 5 deletions docs/configuration/v1.x/metrics/generic-azure-resource.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,17 +14,20 @@ Promitor simplifies defining resource uris by using the subscription & resource

Example:
```yaml
name: demo_generic_queue_size
name: azure_service_bus_active_messages
description: "Amount of active messages of the 'myqueue' queue (determined with Generic provider)"
resourceType: Generic
# Will scrape subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.ServiceBus/namespaces/promitor-messaging
# Where 'sub' & 'rg' are coming from azureMetadata
resourceUri: Microsoft.ServiceBus/namespaces/promitor-messaging
filter: EntityName eq 'orders'
azureMetricConfiguration:
metricName: ActiveMessages
aggregation:
type: Total
resources:
# Will scrape subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.ServiceBus/namespaces/promitor-messaging
# Where 'sub' & 'rg' are coming from azureMetadata
- resourceUri: Microsoft.ServiceBus/namespaces/promitor-messaging
filter: EntityName eq 'orders'
- resourceUri: Microsoft.ServiceBus/namespaces/promitor-messaging
filter: EntityName eq 'items'
```
[&larr; back to metrics declarations](/configuration/metrics)<br />
Expand Down
43 changes: 19 additions & 24 deletions docs/configuration/v1.x/metrics/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ layout: default
title: Metrics Declaration
---

All the Azure Monitor metrics that needs to be scraped are consolidated in one YAML file.
All the Azure Monitor metrics that need to be scraped are consolidated in one YAML file.
This configuration defines the Azure metadata and all the metrics.

# General Declaration
Expand All @@ -20,24 +20,26 @@ As Promitor evolves we need to change the structure of our metrics declaration.

## Metric Defaults

- `metricDefaults.scraping.schedule` - **[REQUIRED]** A cron expression that controls the fequency in which all the configured metrics will be scraped from Azure Monitor. You can use [crontab-generator.org](https://crontab-generator.org/) to generate a cron that fits your needs.
- `metricDefaults.scraping.schedule` - **[REQUIRED]** A cron expression that controls the frequency of which all the configured metrics will be scraped from Azure Monitor. You can use [crontab-generator.org](https://crontab-generator.org/) to generate a cron that fits your needs.
- `metricDefaults.aggregation.interval` - The default interval which defines over what period measurements of a metric should be aggregated.

## Metrics

Every metric that is being declared needs to define the following fields:

- `name` - Name of the metric that will be exposed in the scrape endpoint for Prometheus
- `description` - Description for the metric that will be exposed in the scrape endpoint for Prometheus
- `name` - Name of the metric that will be exposed in the scrape endpoint for Prometheus.
- `description` - Description for the metric that will be exposed in the scrape endpoint for Prometheus.
- `resourceType` - Defines what type of resource needs to be queried.
- `labels` - Defines a set of custom labels to included for a given metric.
- `labels` - Defines a set of custom labels to include for a given metric.
- `azureMetricConfiguration.metricName` - The name of the metric in Azure Monitor to query
- `azureMetricConfiguration.aggregation.type` - The aggregation that needs to be used when querying Azure Monitor
- `azureMetricConfiguration.aggregation.interval` - Overrides the default aggregation interval defined in `metricDefaults.aggregation.interval` with a new interval
- `resources` - An array of one or more resources to get metrics for. The fields required vary
depending on the `resourceType` being created, and are documented for each resource. All resources
support an optional `resourceGroupName` to allow the global resource group to be overridden.

Additionally, the following fields are optional:

- `resourceGroupName` - The resource group to scrape for this metric. This allows you to specify a different resource group from the one configured in `azureMetadata`, enabling you to scrape multiple resource groups with one single configuration.
- `scraping.schedule` - A scraping schedule for the individual metric; overrides the the one specified in `metricDefaults`

## Example
Expand All @@ -56,11 +58,12 @@ metricDefaults:
# Every minute
schedule: "0 * * ? * *"
metrics:
- name: demo_queue_size
description: "Amount of active messages of the 'myqueue' queue"
- name: azure_service_bus_active_messages
description: "The number of active messages on a service bus queue."
resourceType: ServiceBusQueue
namespace: promitor-messaging
queueName: orders
labels:
app: promitor
tier: messaging
scraping:
# Every 2 minutes
schedule: "0 */2 * ? * *"
Expand All @@ -69,20 +72,12 @@ metrics:
aggregation:
type: Total
interval: 00:15:00
- name: demo_queue_dev_size
description: "Amount of active messages of the 'myqueue-dev' queue"
resourceType: ServiceBusQueue
namespace: promitor-messaging-dev
queueName: orders
resourceGroupName: promitor-dev
labels:
app: promitor
stage: dev
azureMetricConfiguration:
metricName: ActiveMessages
aggregation:
type: Total
interval: 00:15:00
resources:
- namespace: promitor-messaging
queueName: orders
- namespace: promitor-messaging-dev
queueName: orders
resourceGroupName: promitor-dev
```
# Supported Azure Services
Expand Down
18 changes: 10 additions & 8 deletions docs/configuration/v1.x/metrics/network-interface.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,16 @@ All supported metrics are documented in the official [Azure Monitor documentatio

Example:
```yaml
- name: demo_azuresnetworkinterface_bytesreceivedrate
description: "Number of bytes the Network Interface sent"
resourceType: NetworkInterface
networkInterfaceName: promitor-network-interface
azureMetricConfiguration:
metricName: BytesReceivedRate
aggregation:
type: Average
name: azure_network_interface_bytes_received_rate
description: "Number of bytes the Network Interface sent"
resourceType: NetworkInterface
azureMetricConfiguration:
metricName: BytesReceivedRate
aggregation:
type: Average
resources:
- networkInterfaceName: promitor-network-interface-1
- networkInterfaceName: promitor-network-interface-2
```
[&larr; back to metrics declarations](/configuration/metrics)<br />
Expand Down
6 changes: 4 additions & 2 deletions docs/configuration/v1.x/metrics/postgresql.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,17 +13,19 @@ All supported metrics are documented in the official [Azure Monitor documentatio

Example:
```yaml
name: postgre_sql_cpu_percent
name: azure_postgre_sql_cpu_percent
description: "The CPU percentage on the server"
resourceType: PostgreSql
serverName: Promitor
scraping:
schedule: "0 */2 * ? * *"
azureMetricConfiguration:
metricName: cpu_percent
aggregation:
type: Average
interval: 00:01:00
resources:
- serverName: Promitor-1
- serverName: Promitor-2
```
[&larr; back to metrics declarations](/configuration/metrics)<br />
Expand Down
6 changes: 4 additions & 2 deletions docs/configuration/v1.x/metrics/redis-cache.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,17 +15,19 @@ You can find more documentation on each metric in the Azure Cache for Redis [mon

Example:
```yaml
name: redis_cache_hits
name: azure_redis_cache_cache_hits
description: "The number of successful key lookups during the specified reporting interval. This maps to keyspace_hits from the Redis INFO command."
resourceType: RedisCache
cacheName: Promitor
scraping:
schedule: "0 */2 * ? * *"
azureMetricConfiguration:
metricName: CacheHits
aggregation:
type: Total
interval: 00:01:00
resources:
- cacheName: Promitor-1
- cacheName: Promitor-2
```
[&larr; back to metrics declarations](/configuration/metrics)<br />
Expand Down
11 changes: 7 additions & 4 deletions docs/configuration/v1.x/metrics/service-bus-queue.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,18 @@ The following scraper-specific metric label(s) will be added:

Example:
```yaml
name: demo_queue_size
description: "Amount of active messages of the 'myqueue' queue"
name: azure_service_bus_queue_active_messages
description: "The number of active messages on a service bus queue"
resourceType: ServiceBusQueue
namespace: promitor-messaging
queueName: orders
azureMetricConfiguration:
metricName: ActiveMessages
aggregation:
type: Total
resources:
- namespace: promitor-messaging
queueName: orders
- namespace: promitor-messaging
queueName: items
```
[&larr; back to metrics declarations](/configuration/metrics)<br />
Expand Down
17 changes: 11 additions & 6 deletions docs/configuration/v1.x/metrics/storage-queue.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,17 +22,22 @@ The following scraper-specific metric label(s) will be added:

Example:
```yaml
name: demo_queue_size
description: "Amount of messages in the 'orders' queue"
name: azure_storage_queue_message_count
description: "The number of messages on an Azure storage queue"
resourceType: StorageQueue
accountName: promitor
queueName: orders
sasToken:
environmentVariable: "SECRETS_STORAGEQUEUE_PROMITOR_SASTOKEN"
azureMetricConfiguration:
metricName: MessageCount
aggregation:
type: Total
resources:
- accountName: promitor
queueName: orders
sasToken:
environmentVariable: "SECRETS_STORAGEQUEUE_PROMITOR_SASTOKEN"
- accountName: promitor
queueName: items
sasToken:
environmentVariable: "SECRETS_STORAGEQUEUE_PROMITOR_SASTOKEN"
```
[&larr; back to metrics declarations](/configuration/metrics)<br />
Expand Down
8 changes: 5 additions & 3 deletions docs/configuration/v1.x/metrics/virtual-machine.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,16 @@ All supported metrics are documented in the official [Azure Monitor documentatio

Example:
```yaml
name: demo_virtualmachine_percentage_cpu
description: "Average percentage cpu usage of our 'promitor-virtual-machine' virtual machine"
name: azure_virtual_machine_percentage_cpu
description: "Average percentage cpu usage on an Azure virtual machine"
resourceType: VirtualMachine
virtualMachineName: promitor-virtual-machine
azureMetricConfiguration:
metricName: Percentage CPU
aggregation:
type: Average
resources:
- virtualMachineName: promitor-virtual-machine-1
- virtualMachineName: promitor-virtual-machine-2
```
[&larr; back to metrics declarations](/configuration/metrics)<br />
Expand Down

0 comments on commit feb8ebc

Please sign in to comment.