Cortex Ingestor PV running out of disk space in a weeks time

[SatyKrish]

We have a cortex setup which supports 8 tenants and ~12 Mil active series of metrics. We have 30 ingestors, each with100gb PV (disk). Longterm metrics are stored in Azure Blob Storage.

We are observing Ingestor PV size is increasing gradually over 2 days and hitting maximum PV capacity soon. We have configured Ingestor to hold blocks for upto 12 hrs. Considering ingestor blocks are pushed to Azure Storage every 2 hours and removed from ingestor every 12 hours (after block expiry), we couldn't understand why ingestor PV size is increasing gradually, even though there is no change in Active series or number of tenants.

Appreciate some help understanding this behavior and guidance on how to plan Ingestor disk capacity.

[SatyKrish]

Any guidance on what could be causing this behaviour? I couldn’t get much insight from logs/metrics.

[jeromeinsf]

Considering ingestor blocks are pushed to Azure Storage every 2 hours and removed from ingestor every 12 hours (after block expiry)

How did you verify this?

[SatyKrish]

@jeromeinsf , I've used Cortex Write dashboard to verify Ingestor pushing blocks to storage every 2 hours, and truncating WAL regularly. However, I don't see a significant decrease in disk utilization.

Active series is between 12-14 mil.

[SatyKrish]

Appreciate some guidance on what could be going wrong? I have tried multiple cortex configurations, but PVs keep growing. We keep expanding to 1TB per PV, and then switch to another instance and rebuild previous instance.

There is not impact, however there is an operational overhead.

[alanprot]

Can you check if the blocks are being shipped looking at the thanos.shipper.json inside of the tenant TSDB?

Also, can u check if u have blocks shipped and older than 12hrs looking at the meta.json files of each block? it should show the min and max time of each block.

Its important to note that if the blocks are not shipped they are NEVER deleted from the disk.

If the blocks are being shipped correctly you should not find blocks older than 12hs on disk (assuming that -blocks-storage.tsdb.retention-period is set to 12 hours)

[SatyKrish]

Unshipped blocks are shown in the cortex_ingester_oldest_unshipped_block_timestamp_seconds metric. Could be related to the below issue, as we have enabled out of order samples.

#5402

Will verify this behaviour with 1.16.0 release.

[SatyKrish]

Upon further investigation we identified many out of order samples being processed. This was due to Prometheus agent on client side running with multiple replicas, whereas Cortex wasn’t configured for HA de-duplication. This resulted in Cortex Ingester spending more time processing duplicate/out of order samples and many blocks remained unshipped.

In the interim we updated Prometheus to run with just 1 replica, and that stopped out of order samples, and this alone prevented persistent volumes from growing.

For longterm, we are considering Grafana agent to run in clustering mode. This offers both high availability and sharding of Prometheus targets.

Thanks for your inputs.