Move default scraping interval to metrics configuration

[tomkerkhove]

Currently, we configure the scraping interval via the `` environment variable. This approach is deprecated and moved to scraping configuration.

With #258 this can be configured per metric and should be used over this value.

Checklist

In order to achieve this we will:

• Update documentation on how to configure this
• Provide clear guidance on breaking change in documentation (depends on Introduce a change log on promitor.io #255)

Metrics Specification

azureMetadata:
  tenantId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
  subscriptionId: yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy
  resourceGroupName: promitor
metricDefaults:
  scraping:
    interval: <interval>
metrics: 
  - name: demo_queue_size
    description: "Amount of active messages of the 'myqueue' queue"
    resourceType: ServiceBusQueue
    namespace: promitor-messaging
    queueName: orders
    azureMetricConfiguration:
      metricName: ActiveMessages
      aggregation:
        type: Total

[brandonh-msft]

I'll take this one too as part of the scraping interval stuff i'm doing

[brandonh-msft]

@tomkerkhove right now we use the IScheduledJob interface to define the recurrence of the scraper's interval. I could see the CronSchedule value of this using metricDefaults.Interval at least as a first pass, but seems like there's quite a bit of re-architecting that needs to take place here.

Initial thought: CronSchedule = 1 minute (not configurable). We then run the scraping job every minute, and look at each metric to determine if it should be run at that time. So there may be minutes where the job kicks off and does nothing and there may be others where it hits multiple metrics. Additionally this means at the first "tick" of the job all configured metrics would get hit.

In any event, if my thoughts above are kosher, it seems like we need to choose a "minimum granularity" for our metrics and that's what CronSchedule should be set to.

Thoughts?

[tomkerkhove]

@brandonh-msft I was rather thinking to schedule an instance of MetricScrapingJob for every configured metric.

This would allow us to:

• Isolate all scrapers from each other and avoid one metric impacting the others
• Be able to make the scraping more straight forward and do all the determination of overridden RG/intervals on the scheduling level
• Make the scraping code a lot more straight forward

I quickly POCed this approach before and it works, but I didn't check if they run sequentially or in parallel. If it doesn't, I'm even wondering if this is something we can contribute back to CronScheduler.AspNetCore.

As of today, I don't really care about configs changing as that's a "change" for me and they should either redeploy or restart the scraper.

What do you think of this approach @brandonh-msft?

[tomkerkhove]

Re-opening as we've only covered configuration, not how we scrape

[brandonh-msft]

Makes sense and just so I understand this would involve:

1. Changing the Builder to parse the config and pull out all metrics & their intervals, properly resolving default metric -> individual metric if no interval is specified on an individual metric.
2. Transforming a TimeSpan declaration of a scraping interval in to a Cron expression so it can be set in to the CronSchedule property of MetricScrapingJob.

Those two pieces together, I think, would enable the approach you're talking about, yes?

[tomkerkhove]

Yes and some refactoring of the internal scraper given it doesn't need to query all metrics anymore.

We already have a dependency on Cronos which should be able to help you!

[brandonh-msft]

@tomkerkhove it doesn't seem like these things are functionally equivalent; i.e. I'm not seeing how one could create a Timespan -> Cron Expression converter given the purpose of Cron is to schedule things and TimeSpan is merely to denote a span of time. This said, today we schedule via Cron; it seems to me that scraping.interval should therefore be a Cron expression and not a TimeSpan - would you agree? perhaps renaming it to scraping.schedule would make this more palatable?

[tomkerkhove]

Perfect idea, like it @brandonh-msft 👍