Question about resource discovery of new & deleted VMs #1243

skuby2 · 2020-08-27T04:39:49Z

skuby2
Aug 27, 2020

Hi. I ran across this software today and am very excited about it for some projects I have. Thank you very much for the great work. The question I have is about VMs that come and go. In my use case I am dynamically spinning up and tearing down VMs. I'm setup to use Prometheus. What I've seen is that new VMs are not shown in Prometheus and Grafana until I restart the promitor-agent-scraper container. Likewise, if promitor-agent-scraper is started while a VM is present and then I delete it, the VM does not go away until I restart promitor-agent-scraper again. In the later case it continues to report old metrics.

Is that the intent for how promitor-agent-scraper and promitor-agent-discovery should behave? Or is the intent (current or eventual) to handle new and deleted VMs, and it just isn't implemented yet?

A bit more info about my setup. Here's a basic outline of a script that shows how I'm running things.
Here are the version of software I'm using:

Prometheus: 2.4.3
promitor-agent-discovery: 0.1.0-preview-1 (which I think is the latest, except for CI builds)
promitor-agent-scraper: 2.0.0-preview-2 (which I think is the latest, except for CI builds)

APP_ID=omit
APP_SECRET=omit
TENANT_ID=omit
SUBSCRIPTION_ID=omit
RESOURCE_GROUP=omit
CONFIG_DIR=~/promitor_configs
SCRAPER_PORT=8999
DISCOVERY_PORT=8998

rm -fr $CONFIG_DIR
mkdir -p $CONFIG_DIR

cat > $CONFIG_DIR/scrape-auth.creds << EOF
PROMITOR_AUTH_APPID=$APP_ID
PROMITOR_AUTH_APPKEY=$APP_SECRET
EOF

cat > $CONFIG_DIR/scrape-metrics.yaml << EOF
version: v1
azureMetadata:
  tenantId: $TENANT_ID
  subscriptionId: $SUBSCRIPTION_ID
  resourceGroupName: $RESOURCE_GROUP
metricDefaults:
  aggregation:
    interval: 00:05:00
  scraping:
    # Every minute
    schedule: "0 * * ? * *"
metrics:
  - name: azure_vm_percentage_cpu
    description: "Average percentage cpu usage on an Azure virtual machine"
    resourceType: VirtualMachine
    azureMetricConfiguration:
      metricName: Percentage CPU
      aggregation:
        type: Average
    resourceDiscoveryGroups:
    - name: vms
  - name: azure_vm_network_in
    description: "The number of bytes received on all network interfaces by the Virtual Machine(s) (Incoming Traffic)"
    resourceType: VirtualMachine
    azureMetricConfiguration:
      metricName: Network In Total
      aggregation:
        type: Average
    resourceDiscoveryGroups:
    - name: vms
  - name: azure_vm_network_out
    description: "The number of bytes out on all network interfaces by the Virtual Machine(s) (Outgoing Traffic)"
    resourceType: VirtualMachine
    azureMetricConfiguration:
      metricName: Network Out Total
      aggregation:
        type: Average
    resourceDiscoveryGroups:
    - name: vms
  - name: azure_vm_outbound_flows
    description: "Outbound Flows are number of current flows in the outbound direction (traffic going out of the VM)"
    resourceType: VirtualMachine
    azureMetricConfiguration:
      metricName: Outbound Flows
      aggregation:
        type: Average
    resourceDiscoveryGroups:
    - name: vms
EOF

cat > $CONFIG_DIR/scrape-runtime.yaml << EOF
server:
  httpPort: 80
metricSinks:
  prometheusScrapingEndpoint:
    metricUnavailableValue: NaN
    enableMetricTimestamps: true # true by default
    baseUriPath: /metrics
metricsConfiguration:
  absolutePath: /config/metrics-declaration.yaml
telemetry:
  applicationInsights:
    instrumentationKey: ABC
    isEnabled: false
    verbosity: warning
  containerLogs:
    isEnabled: true
    verbosity: trace
  defaultVerbosity: trace
azureMonitor:
  logging:
    isEnabled: false
resourceDiscovery:
  host: 172.18.0.1
  port: $DISCOVERY_PORT
EOF

cat > $CONFIG_DIR/discovery-auth.creds << EOF
PROMITOR_DISCOVERY_APPID=$APP_ID
PROMITOR_DISCOVERY_APPSECRET=$APP_SECRET
EOF

cat > $CONFIG_DIR/discovery-declaration.yaml << EOF
version: v1
azureLandscape:
  tenantId: $TENANT_ID
  subscriptions:
  - $SUBSCRIPTION_ID
  cloud: Global
resourceDiscoveryGroups:
- name: vms
  type: VirtualMachine
EOF

cat > $CONFIG_DIR/discovery-runtime.yaml << EOF
server:
  httpPort: 80
telemetry:
  applicationInsights:
    instrumentationKey: ABC
    isEnabled: false
    verbosity: warning
  containerLogs:
    isEnabled: true
    verbosity: trace
  defaultVerbosity: trace
EOF

docker run -d --rm -p $DISCOVERY_PORT:80 --name promitor-agent-resource-discovery                                    \
                         --env PROMITOR_DISCOVERY_APPID=$APP_ID                                                      \
                         --env-file $CONFIG_DIR/discovery-auth.creds                                                 \
                         --volume $CONFIG_DIR/discovery-declaration.yaml:/config/resource-discovery-declaration.yaml \
                         --volume $CONFIG_DIR/discovery-runtime.yaml:/config/runtime.yaml                            \
                         tomkerkhove/promitor-agent-discovery:0.1.0-preview-1

docker run -d --rm -p $SCRAPER_PORT:80 --name promitor-agent-scraper                                                 \
                         --env PROMITOR_AUTH_APPID=$APP_ID                                                           \
                         --env-file $CONFIG_DIR/scrape-auth.creds                                                    \
                         --volume $CONFIG_DIR/scrape-metrics.yaml:/config/metrics-declaration.yaml                   \
                         --volume $CONFIG_DIR/scrape-runtime.yaml:/config/runtime.yaml                               \
                         tomkerkhove/promitor-agent-scraper:2.0.0-preview-2

Answered by skuby2

Aug 27, 2020

Typically, the VMs will be up between 5 and 20 minutes. Ideally Prometheus/Promitor recognizes new VMs within 30 seconds.

I did try reducing the time from 5 minutes to 1 minute to make checking a little faster. I waited 6 minutes before checking. It still appears that things are getting funky, in that new VMs don't show up and old VMs don't go away. It also appears that current VMs are plotting metrics in Grafana that are stale; meaning, the CPU percentage metric stays at the same level and doesn't change, although I'm running some tasks to force the VM CPU usage to spike. Maybe some kind of error is going on that's causing all those issues?

The way I'm checking on the symptoms is just vi…

View full answer

tomkerkhove · 2020-08-27T05:42:05Z

tomkerkhove
Aug 27, 2020
Maintainer

Thanks for trying Promitor and this should definitely be supported!

However, I have some questions:

How often do you spin them up and tier them down?
How fast did you check?

As you can see, Promitor is scheduled to scrape metrics every 5 minutes. For every run, it will discover the resources and scrape does. If all goes well, those VMs should be added/removed in that window.

It's possible that removing will not work correctly so I'll verify that in #1244. However, additions should be done automatically, can you verify that please by starting the scraper, adding a resource and wait a while.

0 replies

skuby2 · 2020-08-27T17:06:52Z

skuby2
Aug 27, 2020
Author

Typically, the VMs will be up between 5 and 20 minutes. Ideally Prometheus/Promitor recognizes new VMs within 30 seconds.

I did try reducing the time from 5 minutes to 1 minute to make checking a little faster. I waited 6 minutes before checking. It still appears that things are getting funky, in that new VMs don't show up and old VMs don't go away. It also appears that current VMs are plotting metrics in Grafana that are stale; meaning, the CPU percentage metric stays at the same level and doesn't change, although I'm running some tasks to force the VM CPU usage to spike. Maybe some kind of error is going on that's causing all those issues?

The way I'm checking on the symptoms is just via Grafana.

4 replies

tomkerkhove Aug 28, 2020
Maintainer

You can certainly reduce the scraping time but Azure Monitor typically takes up to 5 minutes to show metrics so I'm not sure if it will make a difference.

Feel free to keep it running for a while and let me know what you think; I'll dig into it later as well (if I don't forget)

skuby2 Aug 28, 2020
Author

These errors in the prometheus scraper log might point to the issue? I haven't dug through the code, but maybe the scraper believes VMs exist that have actually been deleted. And it's trying to query metrics for those VMs via the Azure API, and it is hanging up and causing the query for the VMs that do actually exist to not succeed.

[19:21:01 INF] Scraping resource collection vms - 08/28/2020 19:21:01 +00:00
[19:21:01 WRN] Failed to log HTTP dependency. Reason: This operation is not supported for a relative URI.
[19:21:01 FTL] Failed to scrape resource collection vms: This instance has already started one or more requests. Properties can only be modified before sending the first request.
System.InvalidOperationException: This instance has already started one or more requests. Properties can only be modified before sending the first request.
   at System.Net.Http.HttpClient.CheckDisposedOrStarted()
   at System.Net.Http.HttpClient.set_BaseAddress(Uri value)
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.SendRequestToApiAsync(HttpRequestMessage request) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 90
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.SendGetRequestAsync(String uri) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 62
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.GetAsync(String resourceDiscoveryGroupName) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 44
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryRepository.GetResourceDiscoveryGroupAsync(String resourceDiscoveryGroupName) in /src/Promitor.Agents.Scraper/ResourceDiscoveryRepository.cs:line 25
   at Promitor.Agents.Scraper.Scheduling.ResourceDiscoveryGroupScrapingJob.ExecuteAsync(CancellationToken cancellationToken) in /src/Promitor.Agents.Scraper/ResourceDiscoveryGroupScrapingJob.cs:line 80
[19:21:01 INF] Scraping resource collection vms - 08/28/2020 19:21:01 +00:00
[19:21:01 WRN] Failed to log HTTP dependency. Reason: This operation is not supported for a relative URI.
[19:21:01 FTL] Failed to scrape resource collection vms: This instance has already started one or more requests. Properties can only be modified before sending the first request.
System.InvalidOperationException: This instance has already started one or more requests. Properties can only be modified before sending the first request.
   at System.Net.Http.HttpClient.CheckDisposedOrStarted()
   at System.Net.Http.HttpClient.set_BaseAddress(Uri value)
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.SendRequestToApiAsync(HttpRequestMessage request) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 90
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.SendGetRequestAsync(String uri) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 62
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.GetAsync(String resourceDiscoveryGroupName) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 44
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryRepository.GetResourceDiscoveryGroupAsync(String resourceDiscoveryGroupName) in /src/Promitor.Agents.Scraper/ResourceDiscoveryRepository.cs:line 25
   at Promitor.Agents.Scraper.Scheduling.ResourceDiscoveryGroupScrapingJob.ExecuteAsync(CancellationToken cancellationToken) in /src/Promitor.Agents.Scraper/ResourceDiscoveryGroupScrapingJob.cs:line 80
[19:21:01 INF] Scraping resource collection vms - 08/28/2020 19:21:01 +00:00
[19:21:01 WRN] Failed to log HTTP dependency. Reason: This operation is not supported for a relative URI.
[19:21:01 FTL] Failed to scrape resource collection vms: This instance has already started one or more requests. Properties can only be modified before sending the first request.
System.InvalidOperationException: This instance has already started one or more requests. Properties can only be modified before sending the first request.
   at System.Net.Http.HttpClient.CheckDisposedOrStarted()
   at System.Net.Http.HttpClient.set_BaseAddress(Uri value)
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.SendRequestToApiAsync(HttpRequestMessage request) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 90
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.SendGetRequestAsync(String uri) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 62
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.GetAsync(String resourceDiscoveryGroupName) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 44
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryRepository.GetResourceDiscoveryGroupAsync(String resourceDiscoveryGroupName) in /src/Promitor.Agents.Scraper/ResourceDiscoveryRepository.cs:line 25
   at Promitor.Agents.Scraper.Scheduling.ResourceDiscoveryGroupScrapingJob.ExecuteAsync(CancellationToken cancellationToken) in /src/Promitor.Agents.Scraper/ResourceDiscoveryGroupScrapingJob.cs:line 80
[19:21:01 INF] Scraping resource collection vms - 08/28/2020 19:21:01 +00:00
[19:21:01 WRN] Failed to log HTTP dependency. Reason: This operation is not supported for a relative URI.
[19:21:01 FTL] Failed to scrape resource collection vms: This instance has already started one or more requests. Properties can only be modified before sending the first request.
System.InvalidOperationException: This instance has already started one or more requests. Properties can only be modified before sending the first request.
   at System.Net.Http.HttpClient.CheckDisposedOrStarted()
   at System.Net.Http.HttpClient.set_BaseAddress(Uri value)
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.SendRequestToApiAsync(HttpRequestMessage request) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 90
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.SendGetRequestAsync(String uri) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 62
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.GetAsync(String resourceDiscoveryGroupName) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 44
   at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryRepository.GetResourceDiscoveryGroupAsync(String resourceDiscoveryGroupName) in /src/Promitor.Agents.Scraper/ResourceDiscoveryRepository.cs:line 25
   at Promitor.Agents.Scraper.Scheduling.ResourceDiscoveryGroupScrapingJob.ExecuteAsync(CancellationToken cancellationToken) in /src/Promitor.Agents.Scraper/ResourceDiscoveryGroupScrapingJob.cs:line 80
[19:21:01 INF] Request starting HTTP/1.1 GET http://172.18.0.1:8999/metrics
[19:21:01 VRB] All hosts are allowed.
[19:21:01 DBG] No candidates found for the request path '/metrics'
[19:21:01 DBG] Request did not match any endpoints
[19:21:01 VRB] Setting correlation response header 'X-Transaction-ID' to '69367be3-5d0a-49cc-b2ce-46a4f455d514'
[19:21:01 VRB] Setting correlation response header 'RequestId' to '0HM2BFKU0TD4J:00000008'
[19:21:01 INF] HTTP Request GET http://172.18.0.1:8999//metrics completed with 200 in 00:00:00.0004934 at 08/28/2020 19:21:01 +00:00 - (Context: {"Accept": ["text/plain;version=0.0.4;q=1,*/*;q=0.1"], "Accept-Encoding": ["gzip"], "Host": ["172.18.0.1:8999"], "User-Agent": ["Prometheus/2.4.3"], "X-Prometheus-Scrape-Timeout-Seconds": ["10.000000"]})
[19:21:01 INF] Request finished in 1.0204ms 200 text/plain; version=0.0.4

tomkerkhove Aug 29, 2020
Maintainer

This is the same issue as #1247 which will be fixed in the next preview; sorry about that!

tomkerkhove Sep 11, 2020
Maintainer

Available here: https://github.com/tomkerkhove/promitor/releases/tag/2.0.0-preview-3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Question about resource discovery of new & deleted VMs #1243

{{title}}

Replies: 2 comments 4 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Question about resource discovery of new & deleted VMs #1243

skuby2 Aug 27, 2020

Replies: 2 comments · 4 replies

tomkerkhove Aug 27, 2020 Maintainer

skuby2 Aug 27, 2020 Author

tomkerkhove Aug 28, 2020 Maintainer

skuby2 Aug 28, 2020 Author

tomkerkhove Aug 29, 2020 Maintainer

tomkerkhove Sep 11, 2020 Maintainer

skuby2
Aug 27, 2020

Replies: 2 comments 4 replies

tomkerkhove
Aug 27, 2020
Maintainer

skuby2
Aug 27, 2020
Author

tomkerkhove Aug 28, 2020
Maintainer

skuby2 Aug 28, 2020
Author

tomkerkhove Aug 29, 2020
Maintainer

tomkerkhove Sep 11, 2020
Maintainer