Receive: Add parameter to set out-of-order time window

[matej-g]

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

Although we recently updated Prometheus version that experimentally allows to ingest out-of-order samples, our users have no way of setting this as of now. This change adds a parameter to set this time window. In addition it also handles a new type of error (sample too old) that can occur if this feature is enabled.

Verification

Manually tried 😶

[yeya24]

Overall this looks good.

[yeya24]

Should we also add flag for OutOfOrderCapMax?

[saswatamcode]

Looks good overall!

[yeya24]

This should fix #2599 and #2114 🎉

[matej-g]

Should we also add flag for OutOfOrderCapMax?

I can add that as well, wasn't too sure if this is something end users will need to tweak ordinarily.

[yeya24]

LGTM

[yeya24]

@matej-g I just realized that one more thing. On the shipper side (the component who is responsible for uploading blocks), there is also a overlapping checker.
Do you know if OOO samples produce blocks that is overlapped and also compaction level 2?
This will cause problems since the checker will fail.

[matej-g]

@matej-g I just realized that one more thing. On the shipper side (the component who is responsible for uploading blocks), there is also a overlapping checker. Do you know if OOO samples produce blocks that is overlapped and also compaction level 2? This will cause problems since the checker will fail.

So, I'm not super familiar with the compaction in Prometheus, but as far as I can understand the code:

• When "normal" head is compacted, also the OOO head gets compacted - so these blocks at compaction level 1 will overlap (but this should be expected)
• Once these normal and OOO blocks are compacted further, however, I think there should be no overlap at level 2

So it should not be an issue. Does this make sense @yeya24?

[matej-g]

@yeya24 if I now look better at the shipper code, I notice we don't ship blocks with > 1 compaction level, so the overlap check should not even kick in. But it makes me wonder why we even run TSDB compaction on receivers if we basically ignore blocks with level > 1 🤔. Scratch that, since compaction in TSDB also takes care of head compaction, I still wonder though why run block compaction.

[yeya24]

I notice we don't ship blocks with > 1 compaction level, so the overlap check should not even kick in

Yeah I didn't know how OOO really works so not sure if it produces a level 1 overlapped block compacted or it creates 2 level 1 blocks with a normal one and 1 OOO one.

If it creates 2 level 1 block with a normal one and 1 OOO. Do you know does Prometheus will compact the two if there is overlap?

[matej-g]

Got you now, I see we don't run compaction because of min == max block duration, yeah we should be safe, thanks for checking 👍