Expectation of working when Target Allocator is down

[sfc-gh-akrishnan]

When Target Allocator is unreachable or down for some duration, what is the expectation of working from the Otel-Collector's perspective.

Assumption:

• All the otel-collectors pods have gotten the configuration of targets to scrape from at least ONCE

Possible work-patterns:

• Continue to scrape endpoints with the previously gotten config
• Stop scraping till target allocator again becomes available for scrapping

TIA : )

[sfc-gh-akrishnan]

Hi Folks,
Requesting some guidance on the item. Thank you :)

[sfc-gh-akrishnan]

@jaronoff97, bringing this to your attention if you could help me on this regard or tag the right people who can answer the question?

TIA

[jaronoff97]

to answer the immediate question: we recommend running the target allocator with the consistent hashing filter strategy with at least 2 replicas to enable a high availability mode in the scenario where the target allocator is down. For now there is no fallback in the collector as that would potentially cause dual scrapes. This is a design choice that was made with the CAP theorem in mind. i.e. imagine there's a network partition between the collectors and target allocators, we opt for the collectors to remain available if they are handling other workloads (tracing and logs) and correct (theyre not sending incorrect metric data). Basically, we are doing the second approach you mentioned where we just stop scraping because we would fail those scrapes.

If you have more questions by the way, our group is a bit more responsive on slack if there's more urgency to your questions.