Scraping resources regularly failing

[bluepixbe]

In combination with the resource discovery agent we see multiple error like the following in the scraper agent logs:
[12:12:00 FTL] Failed to scrape resource collection vm-landscape: This instance has already started one or more requests. Properties can only be modified before sending the first request.

Btw. really nice feature this resource discovery agent

Expected Behavior

No such errors as it seems to break the fetching of the metrics.

Actual Behavior

[12:21:01 INF] Found value 100 for metric network_expressroutecircuits_arpavailability with aggregation interval 00:05:00
[12:21:01 INF] Found value 100 for metric network_expressroutecircuits_bgpavailability with aggregation interval 00:05:00
[12:21:01 INF] Metrics declaration is using spec version v1
[12:21:01 INF] Metrics declaration is using spec version v1
[12:21:08 INF] Request starting HTTP/1.1 GET http://172.17.0.3:80/metrics
[12:21:08 INF] Request finished in 0.2773ms 200 text/plain; version=0.0.4
[12:21:23 INF] Request starting HTTP/1.1 GET http://172.17.0.3:80/metrics
[12:21:23 INF] Request finished in 0.3357ms 200 text/plain; version=0.0.4
[12:21:38 INF] Request starting HTTP/1.1 GET http://172.17.0.3:80/metrics
[12:21:38 INF] Request finished in 0.3034ms 200 text/plain; version=0.0.4
[12:21:53 INF] Request starting HTTP/1.1 GET http://172.17.0.3:80/metrics
[12:21:53 INF] Request finished in 0.3079ms 200 text/plain; version=0.0.4
[12:22:00 INF] Scraping Azure Monitor - 08/27/2020 12:22:00 +00:00
[12:22:00 INF] Scraping network_expressroutecircuits_arpavailability for resource type Generic
[12:22:00 INF] Scraping resource collection vm-landscape - 08/27/2020 12:22:00 +00:00
[12:22:00 WRN] Failed to log HTTP dependency. Reason: This operation is not supported for a relative URI.
[12:22:00 FTL] Failed to scrape resource collection vm-landscape: This instance has already started one or more requests. Properties can only be modified before sending the first request.
System.InvalidOperationException: This instance has already started one or more requests. Properties can only be modified before sending the first request.
at System.Net.Http.HttpClient.CheckDisposedOrStarted()
at System.Net.Http.HttpClient.set_BaseAddress(Uri value)
at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.SendRequestToApiAsync(HttpRequestMessage request) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 72
at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.SendGetRequestAsync(String uri) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 57
at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.GetAsync(String resourceDiscoveryGroupName) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 40
at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryRepository.GetResourceDiscoveryGroupAsync(String resourceDiscoveryGroupName) in /src/Promitor.Agents.Scraper/ResourceDiscoveryRepository.cs:line 23
at Promitor.Agents.Scraper.Scheduling.ResourceDiscoveryGroupScrapingJob.ExecuteAsync(CancellationToken cancellationToken) in /src/Promitor.Agents.Scraper/ResourceDiscoveryGroupScrapingJob.cs:line 80
[12:22:00 INF] Scraping Azure Monitor - 08/27/2020 12:22:00 +00:00
[12:22:00 INF] Scraping network_expressroutecircuits_bgpavailability for resource type Generic
[12:22:01 INF] Found value 100 for metric network_expressroutecircuits_arpavailability with aggregation interval 00:05:00
[12:22:01 INF] Metrics declaration is using spec version v1
[12:22:02 INF] Found value 100 for metric network_expressroutecircuits_bgpavailability with aggregation interval 00:05:00
[12:22:02 INF] Metrics declaration is using spec version v1
[12:22:08 INF] Request starting HTTP/1.1 GET http://172.17.0.3:80/metrics
[12:22:08 INF] Request finished in 0.3236ms 200 text/plain; version=0.0.4
[12:22:23 INF] Request starting HTTP/1.1 GET http://172.17.0.3:80/metrics
[12:22:23 INF] Request finished in 0.3189ms 200 text/plain; version=0.0.4

Steps to Reproduce the Problem

1. No specific steps. See configs below. Just starting both agents and we get the errors

Configuration

Used scraping configuration

....
metrics:
  - name: azure_virtual_machine_percentage_cpu
    description: "Average percentage cpu usage on an Azure virtual machine"
    resourceType: VirtualMachine
    scraping:
      # Every 2 minutes
      schedule: "0 */2 * ? * *"
    azureMetricConfiguration:
      metricName: Percentage CPU
      aggregation:
        type: Average
        interval: 00:01:00
    resourceDiscoveryGroups:
    - name: vm-landscape
  - name: network_expressroutecircuits_arpavailability
    description: ""
    resourceType: Generic
    azureMetricConfiguration:
      metricName: ArpAvailability
      aggregation:
        type: Average
    resources:
    - resourceUri: Microsoft.Network/expressRouteCircuits/NAME
  - name: network_expressroutecircuits_bgpavailability
    description: ""
    resourceType: Generic
    azureMetricConfiguration:
      metricName: BgpAvailability
      aggregation:
        type: Average
    resources:
    - resourceUri: Microsoft.Network/expressRouteCircuits/NAME

Used discovery configuration

version: v1
azureLandscape:
  tenantId: ID
  subscriptions:
  - SUBID
  cloud: Global
resourceDiscoveryGroups:
- name: vm-landscape
  type: VirtualMachine

Specifications

• Version: 2.0.0-preview-2 & 0.1.0-preview-1
• Platform: Docker
• Subsystem:

[tomkerkhove]

Yes, this is a nasty one but was already fixed in a next release! (see #1235)

[tomkerkhove]

Will ship a new preview ASAP

[bluepixbe]

Awesome. Thanks for the fast reply.

[tomkerkhove]

I'll close it since it's done but thanks for the report anyway!

Feel free to re-open if next preview still shows this and sorry for the inconvenience!

[bluepixbe]

@tomkerkhove Do you have any plans to release the next preview in the next couple of days?

[tomkerkhove]

I'm wrapping up changes to Helm chart and will then most likely ship preview 3 next week.

Hope that's OK?

[bluepixbe]

@tomkerkhove That would perfectly fit. Thanks!

[tomkerkhove]

Available here: https://github.com/tomkerkhove/promitor/releases/tag/2.0.0-preview-3