-
-
Notifications
You must be signed in to change notification settings - Fork 92
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Documentation for runtime configuration via YAML (#643)
* Move Azure Monitor configuration to dedicated page * WIP * WIP * Move Azure Monitor configuration to dedicated page * WIP * WIP * WIP * Add metricUnavailableValue & enableMetricTimestamps * Use dedicated folders * Add old landing page Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com> * Finalize documentation * Use v0.X docs for now * Fix broken link * Use Azure Monitor v1.0 docs * Change intro & fix broken link
- Loading branch information
1 parent
da6b159
commit e7c6669
Showing
18 changed files
with
341 additions
and
16 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| --- | ||
| layout: default | ||
| title: Authentication with Azure Monitor | ||
| --- | ||
|
|
||
| Here is an overview of how you configure how Promitor integrates with Azure Monitor. | ||
|
|
||
| # Authentication with Azure Monitor | ||
| Authentication with Azure Monitor is fully integrated with Azure AD. In order to use Promitor, you'll need to [create an Azure AD Application](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#create-an-azure-active-directory-application), that will be used for integrating with the Azure Monitor API. | ||
|
|
||
| The following environment variables need to be provided: | ||
| - **PROMITOR_AUTH_APPID** - Id of the Azure AD entity to authenticate with | ||
| - **PROMITOR_AUTH_APPKEY** - Secret of the Azure AD entity to authenticate with | ||
|
|
||
| This information can be found on the newly created AD Application as documented [here](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#get-application-id-and-authentication-key). | ||
|
|
||
| The entity in the Azure AD needs to have `Monitoring Reader` permission on the resource group that will be queried. More information can be found [here](https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitoring-roles-permissions-security). | ||
|
|
||
| [← back](/) |
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| --- | ||
| layout: default | ||
| title: Generic Azure Resource Declaration | ||
| --- | ||
|
|
||
| ## Generic Azure Resource -  | ||
| You can declare to scrape a generic Azure resource via the `Generic` resource type. | ||
|
|
||
| The following fields need to be provided: | ||
| - `resourceUri` - The uri of the Azure resource to scrape. | ||
| - `filter` - The filter to use to have fine-grained metrics. Example: `EntityName eq 'orders'` | ||
|
|
||
| Promitor simplifies defining resource uris by using the subscription & resource group defined in `azureMetadata` so that your configuration is small & readable. | ||
|
|
||
| Example: | ||
| ```yaml | ||
| name: demo_generic_queue_size | ||
| description: "Amount of active messages of the 'myqueue' queue (determined with Generic provider)" | ||
| resourceType: Generic | ||
| # Will scrape subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.ServiceBus/namespaces/promitor-messaging | ||
| # Where 'sub' & 'rg' are coming from azureMetadata | ||
| resourceUri: Microsoft.ServiceBus/namespaces/promitor-messaging | ||
| filter: EntityName eq 'orders' | ||
| azureMetricConfiguration: | ||
| metricName: ActiveMessages | ||
| aggregation: | ||
| type: Total | ||
| ``` | ||
| [← back to metrics declarations](/configuration/metrics)<br /> | ||
| [← back to introduction](/) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,106 @@ | ||
| --- | ||
| layout: default | ||
| title: Metrics Declaration | ||
| --- | ||
|
|
||
| All the Azure Monitor metrics that needs to be scraped are consolidated in one YAML file. | ||
| This configuration defines the Azure metadata and all the metrics. | ||
|
|
||
| # General Declaration | ||
|
|
||
| As Promitor evolves we need to change the structure of our metrics declaration. | ||
|
|
||
| `version: {version}` - **[REQUIRED]** Version of declaration that is used. Allowed values are `v1`. | ||
|
|
||
| ## Azure | ||
|
|
||
| - `azureMetadata.tenantId` - The id of the Azure tenant that will be queried. | ||
| - `azureMetadata.subscriptionId` - The id of the default subscription to query. | ||
| - `azureMetadata.resourceGroupName` - The name of the default resource group to query. | ||
|
|
||
| ## Metric Defaults | ||
|
|
||
| - `metricDefaults.scraping.schedule` - **[REQUIRED]** A cron expression that controls the fequency in which all the configured metrics will be scraped from Azure Monitor. You can use [crontab-generator.org](https://crontab-generator.org/) to generate a cron that fits your needs. | ||
| - `metricDefaults.aggregation.interval` - The default interval which defines over what period measurements of a metric should be aggregated. | ||
|
|
||
| ## Metrics | ||
|
|
||
| Every metric that is being declared needs to define the following fields: | ||
|
|
||
| - `name` - Name of the metric that will be exposed in the scrape endpoint for Prometheus | ||
| - `description` - Description for the metric that will be exposed in the scrape endpoint for Prometheus | ||
| - `resourceType` - Defines what type of resource needs to be queried. | ||
| - `labels` - Defines a set of custom labels to included for a given metric. | ||
| - `azureMetricConfiguration.metricName` - The name of the metric in Azure Monitor to query | ||
| - `azureMetricConfiguration.aggregation.type` - The aggregation that needs to be used when querying Azure Monitor | ||
| - `azureMetricConfiguration.aggregation.interval` - Overrides the default aggregation interval defined in `metricDefaults.aggregation.interval` with a new interval | ||
|
|
||
| Additionally, the following fields are optional: | ||
|
|
||
| - `resourceGroupName` - The resource group to scrape for this metric. This allows you to specify a different resource group from the one configured in `azureMetadata`, enabling you to scrape multiple resource groups with one single configuration. | ||
| - `scraping.schedule` - A scraping schedule for the individual metric; overrides the the one specified in `metricDefaults` | ||
|
|
||
| ## Example | ||
| Here is an example of how you can scrape two Azure Service Bus queues in different resource groups, one in the `promitor` resource group and one on the `promitor-dev` resource group: | ||
|
|
||
| ```yaml | ||
| version: v1 | ||
| azureMetadata: | ||
| tenantId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | ||
| subscriptionId: yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy | ||
| resourceGroupName: promitor | ||
| metricDefaults: | ||
| aggregation: | ||
| interval: 00:05:00 | ||
| scraping: | ||
| # Every minute | ||
| schedule: "0 * * ? * *" | ||
| metrics: | ||
| - name: demo_queue_size | ||
| description: "Amount of active messages of the 'myqueue' queue" | ||
| resourceType: ServiceBusQueue | ||
| namespace: promitor-messaging | ||
| queueName: orders | ||
| scraping: | ||
| # Every 2 minutes | ||
| schedule: "0 */2 * ? * *" | ||
| azureMetricConfiguration: | ||
| metricName: ActiveMessages | ||
| aggregation: | ||
| type: Total | ||
| interval: 00:15:00 | ||
| - name: demo_queue_dev_size | ||
| description: "Amount of active messages of the 'myqueue-dev' queue" | ||
| resourceType: ServiceBusQueue | ||
| namespace: promitor-messaging-dev | ||
| queueName: orders | ||
| resourceGroupName: promitor-dev | ||
| labels: | ||
| app: promitor | ||
| stage: dev | ||
| azureMetricConfiguration: | ||
| metricName: ActiveMessages | ||
| aggregation: | ||
| type: Total | ||
| interval: 00:15:00 | ||
| ``` | ||
| # Supported Azure Services | ||
| [Generic Azure Resource](generic-azure-resource) allows you to scrape every Azure service supported by Azure Monitor. | ||
| We also provide a simplified way to scrape the following Azure resources: | ||
| - [Azure Cache for Redis](redis-cache) | ||
| - [Azure Container Instances](container-instances) | ||
| - [Azure Container Registry](container-registry) | ||
| - [Azure Cosmos DB](cosmos-db) | ||
| - [Azure Database for PostgreSQL](postgresql) | ||
| - [Azure Network Interface](network-interface) | ||
| - [Azure Service Bus Queue](service-bus-queue) | ||
| - [Azure Storage Queue](storage-queue) | ||
| - [Azure Virtual Machine](virtual-machine) | ||
| Want to help out? Create an issue and [contribute a new scraper](https://github.com/tomkerkhove/promitor/blob/master/adding-a-new-scraper.md). | ||
| [← back](/) |
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| --- | ||
| layout: default | ||
| title: Azure Service Bus Queue Declaration | ||
| --- | ||
|
|
||
| ## Azure Service Bus Queue -  | ||
| You can declare to scrape an Azure Service Bus Queue via the `ServiceBusQueue` resource type. | ||
|
|
||
| The following fields need to be provided: | ||
| - `namespace` - The name of the Service Bus namespace | ||
| - `queueName` - The name of the queue | ||
|
|
||
| Supported metrics: | ||
| - `IncomingMessages` | ||
| - `IncomingRequests` | ||
| - `ActiveMessages` | ||
| - `Messages` | ||
| - `Size` | ||
|
|
||
| Example: | ||
| ```yaml | ||
| name: demo_queue_size | ||
| description: "Amount of active messages of the 'myqueue' queue" | ||
| resourceType: ServiceBusQueue | ||
| namespace: promitor-messaging | ||
| queueName: orders | ||
| azureMetricConfiguration: | ||
| metricName: ActiveMessages | ||
| aggregation: | ||
| type: Total | ||
| ``` | ||
| [← back to metrics declarations](/configuration/metrics)<br /> | ||
| [← back to introduction](/) |
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,146 @@ | ||
| --- | ||
| layout: default | ||
| title: Runtime Configuration | ||
| --- | ||
|
|
||
| This article covers an overview of all the knobs that you can tweak to align the runtime with your needs. | ||
|
|
||
| Promitor runtime is configured by mounting a volume to `/config/runtime.yaml`. | ||
|
|
||
| Here is a complete example: | ||
|
|
||
| ```yaml | ||
| server: | ||
| httpPort: 80 # Optional. Default: 80 | ||
| prometheus: | ||
| metricUnavailableValue: NaN # Optional. Default: NaN | ||
| enableMetricTimestamps: false # Optional. Default: true | ||
| scrapeEndpoint: | ||
| baseUriPath: /metrics # Optional. Default: /metrics | ||
| metricsConfiguration: | ||
| absolutePath: /config/metrics-declaration.yaml # Optional. Default: /config/metrics-declaration.yaml | ||
| telemetry: | ||
| applicationInsights: | ||
| instrumentationKey: ABC # Optional. Note: Required to be specified when turned on | ||
| isEnabled: false # Optional. Default: false | ||
| verbosity: trace # Optional. Default: N/A | ||
| containerLogs: | ||
| isEnabled: true # Optional. Default: true | ||
| verbosity: trace # Optional. Default: N/A | ||
| defaultVerbosity: error # Optional. Default: error | ||
| ``` | ||
| _Note: Using Promitor v0.x? [Use environment variables](./../v0.x/) to configure the runtime._ | ||
| # Runtime | ||
| The Promitor runtime is flexible and allows you to configure it to meet your needs: | ||
| - `server.httpPort` - Defines the port to serve HTTP traffic _(default 80)_ | ||
|
|
||
| Example: | ||
| ```yaml | ||
| server: | ||
| httpPort: 80 # Optional. Default: 80 | ||
| ``` | ||
|
|
||
| # Scraping | ||
| Promitor automatically scrapes Azure Monitor and makes the information available based on the metrics configuration. | ||
|
|
||
| The behavior of this can be configured to fit your needs: | ||
| - `prometheus.metricUnavailableValue` - Defines the value that will be reported if a metric is unavailable. (Default: `NaN`) | ||
| - `prometheus.enableMetricTimestamps` - Defines whether or not a timestamp should be included when the value was scraped on Azure Monitor. Supported values are `True` to opt-in & `False` to opt-out. (Default: `true`) | ||
| - `prometheus.scrapeEndpoint.baseUriPath` - Controls the path where the scraping endpoint for Prometheus is being exposed. (Default: `/metrics`) | ||
|
|
||
| Example: | ||
| ```yaml | ||
| prometheus: | ||
| metricUnavailableValue: NaN # Optional. Default: NaN | ||
| enableMetricTimestamps: false # Optional. Default: true | ||
| scrapeEndpoint: | ||
| baseUriPath: /metrics # Optional. Default: /metrics | ||
| ``` | ||
|
|
||
| # Metric Configuration | ||
| Promitor will scrape the Azure Monitor metrics that are configured via a metric declaration YAML. | ||
|
|
||
| The behavior of this is configurable: | ||
|
|
||
| - `metricsConfiguration.absolutePath` - Defines the location of the YAML file that declares what Azure Monitor metrics to scrape. (Default: `/config/metrics-declaration.yaml`) | ||
|
|
||
| Example: | ||
| ```yaml | ||
| metricsConfiguration: | ||
| absolutePath: /config/metrics-declaration.yaml # Optional. Default: /config/metrics-declaration.yaml | ||
| ``` | ||
|
|
||
| # Telemetry | ||
| We provide insights in how our runtime is doing and is written to one or more sinks. | ||
|
|
||
| You can determine what telemetry sinks you want and what the default verbosity should be via the runtime YAML. | ||
|
|
||
| General telemetry information can be configured: | ||
| - `telemetry.defaultVerbosity`- Defines the default minimum log level that should be logged if a sink does not provide one. Allowed values are `Trace`, `Debug`, `Information`, `Warning`, `Error`, `Critical`, `None` ordered from most to least verbose. (Default: `Error`) | ||
|
|
||
| To learn more about the configured sinks and their configuration, see [**"Telemetry Sinks"**](#telemetry-sinks). | ||
|
|
||
| Example: | ||
|
|
||
| ```yaml | ||
| telemetry: | ||
| applicationInsights: | ||
| # [...] | ||
| containerLogs: | ||
| # [...] | ||
| defaultVerbosity: error # Optional. Default: error | ||
| ``` | ||
|
|
||
| ## Telemetry Sinks | ||
| Promitor provides the telemetry, but it's up to you to choose where you want to send it to. | ||
|
|
||
| We currently support the following sinks: | ||
| - [**Container Logs** (stdout/stderr)](#container-logs) | ||
| - [**Azure Application Insights**](#azure-application-insights) | ||
|
|
||
| ### Container Logs | ||
| Promitor can send telemetry to `stdout`/`stderr`. | ||
|
|
||
| In order to enable use this sink, the following configuration needs to be provided: | ||
| - `telemetry.containerLogs.isEnabled` - Determines if the sink is used or not. (Default: `true`) | ||
| - `telemetry.containerLogs.verbosity` - Verbosity to use for this sink, if not specified then the `telemetry.defaultVerbosity` will be used. (Optional) | ||
|
|
||
| Example: | ||
|
|
||
| ```yaml | ||
| telemetry: | ||
| containerLogs: | ||
| isEnabled: true # Optional. Default: true | ||
| verbosity: trace # Optional. Default: N/A | ||
| defaultVerbosity: error # Optional. Default: error | ||
| ``` | ||
|
|
||
| ### Azure Application Insights | ||
| Promitor can send telemetry to Azure Application Insights when there is a need to. | ||
|
|
||
| It currently supports: | ||
| - Exceptions during scraping | ||
|
|
||
| In order to enable use this sink, the following configuration needs to be provided: | ||
|
|
||
| - `telemetry.applicationInsights.isEnabled` - Determines if the sink is used or not. (Default: `true`) | ||
| - `telemetry.applicationInsights.verbosity` - Verbosity to use for this sink, if not specified then the `telemetry.defaultVerbosity` will be used. (Optional) | ||
| - `telemetry.applicationInsights.instrumentationKey` - Defines the instrumentation key to use when sending telemetry to Azure Application Insights | ||
|
|
||
| Example: | ||
|
|
||
| ```yaml | ||
| telemetry: | ||
| applicationInsights: | ||
| instrumentationKey: ABC # Optional. Note: Required to be specified when turned on | ||
| isEnabled: false # Optional. Default: false | ||
| verbosity: trace # Optional. Default: N/A | ||
| containerLogs: | ||
| isEnabled: true # Optional. Default: true | ||
| verbosity: trace # Optional. Default: N/A | ||
| defaultVerbosity: error # Optional. Default: error | ||
| ``` | ||
|
|
||
| [← back](/) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters