Seeing lots of "Out of Order" after restarts

[gouthamve]

Asked in Slack here: https://cloud-native.slack.com/archives/CCYDASBLP/p1607631100234900

Hi all.

My Cortex instance went down for 30 minutes today. After I can get Cortex up and running, I am seen many errors of out of order sample on prometheus:

prd-ath-barimetrics-distributor-d5b545794-ptr4t distributor level=error ts=2020-12-10T20:07:13.52734165Z caller=push.go:47 org_id=0 traceID=2415039e6f8f2e53 msg="push error" err="rpc error: code = Code(400) desc = user=0: series={__name__=\"iib_requests_duration_seconds_bucket\", canal=\"plataforma\", component_name=\"nrp-api\", component_version=\"1.0.26\", env=\"piloto\", exported_component_name=\"nrp-api\", instance=\"pel0pbb1p018.dispositivos.bb.com.br:8025\", job=\"nrp-api iib\", le=\"+Inf\", prometheus=\"ath-barimetrics-monitor-plat-nrp-api/prd-ath-barimetrics-monitor-plat-nr\", prometheus_replica=\"prometheus-prd-ath-barimetrics-monitor-plat-nr-0\", server_name=\"nrp-api-1\", status=\"2xx\", target=\"IIB\", transaction=\"4565803\", transport=\"Rendezvous\"}, timestamp=2020-12-10T20:06:32.039Z: out of order sample"

How can I solve it?

[gouthamve]

These errors are expected and should be "ok". We do have a long standing issue where we are tracking it and it has more information as to why these errors crop up: #670