Add option to stop reporting deleted resources in Prometheus scrape endpoint

[nikosmeds]

Proposal

Add a setting to stop reporting deleted resources in Prometheus scrape endpoint.

From https://docs.promitor.io/faq#how-does-promitor-handle-deleted-resources

Removed resources will immediately stop being scraped by Promitor, but still be reported in Prometheus scrape endpoint.

Reporting metrics for deleted resources:

• can be misleading to those viewing the metrics (e.g. this is not standard behaviour for most exporters)
• leads to false-positive alerts with promitor_scrape_error metric, as the metric returns 1 for any deleted resources
• unnecessarily stores additional time-series in Prometheus

An option to stop reporting metrics for deleted resources would resolve the above issues.

Component

Scraper

Contact Details

No response

[nikosmeds]

Btw, thanks for Promitor @tomkerkhove - it's really helpful. I understand the FAQ recommends we enable Prometheus metric timestamps - however, this is not an option for us.

A few hours after a resource is deleted, our Prometheus systems with remote-write begin reporting err: out of bounds due to reported time being less than valid time.

[tomkerkhove]

Thank YOU for using Promitor! Always happy to hear feedback or list you as an end-user.

I'm not sure how I can fix this, though, as once it's reported it is still around. I will ask in our depending library.

Update: prom-client-net/prom-client#177

[tomkerkhove]

This is coming as per prom-client-net/prom-client#177 and prom-client-net/prom-client#178 so then we can do a more proper clean-up.

However, we would need to check if this is an opt-in or not given others who use timestamps might want to keep them around.

[tomkerkhove]

This is now supported (see prom-client-net/prom-client#178 & prom-client-net/prom-client#177)

[tomkerkhove]

This should be done through a feature flag where the default should be to just keep the metric; to avoid breaking changes.

[phnx47]

First, need to upgrade to latest versions of Prometheus.Client packages #1772

[tomkerkhove]

Yup