Key space is not cleanup up by retention policy

[ahermspacketwerk]

affected version: influx 1.3.1

We are trying to use influxdb in a memory limited environment. Our data scheme requires to assign a medium number (1000 to 10000) of tag values.

We already found out that this is not optimal regarding the memory usage of influxdb.

Our current attempt is to limit the number of series with max-series-per-database = 10000.
So far, it works as expected. Whenever we hit the limit, new tag values are dropped during import, which is what we try to achieve.

But the key-space of the tag values seems to be not released by the retention policies. Even when the series should have been dropped we cannot insert new entries.
As a result the database looks empty (doing a select * query) but we cannot insert new elements.

This issue can be solved by restarting influxd. After restart we can insert new tag values again.

Any idea how we can overcome this?

[e-dard]

Thanks for the @ahermspacketwerk,

Could you just check that this issue occurs on 1.3.5? It sounds like the index isn't being updated appropriately when the retention policy service drops a shard. We should be checking if we have dropped the last existing point for a series, but possibly that's being missed somewhere.

[ahermspacketwerk]

Sorry for the late response. I was out of office and it take some time until we have collected enough data in our system to hit the limit.

Anyway - yes, the same problem occurs with InfluxDB v1.3.5 (git: HEAD 9d90010)

[e-dard]

Thanks @ahermspacketwerk. I'm treating this as a bug and we'll work on a fix for the next release.

[e-dard]

Hi @ahermspacketwerk

I'm having trouble reproducing this, and the code path followed by deleting a retention policy seems to be the correct one.

Can you confirm in your logs that you're seeing lines like retention policy shard deletion check commencing, and also, are you seeing any of the following lines?

failed to delete shard group ?? from database ??, retention policy ??
deleted shard group ?? from database ??, retention policy ??
failed to delete shard ID ?? from database ??, retention policy ??: ??
shard ID ?? from database ??, retention policy ??, deleted
error pruning shard groups: ??

If you see any of those would you be able to provide your logs? You can email them to me privately if you don't want to post them on the ticket. My email is edd@<name_of_database>.com

[ahermspacketwerk]

Hi @e-dard

I will reproduce the problem on our setup and look into the log files. Once, the problem is established, we could take a closer look at the situation. We have seen weird effects like shards that are present, but the tables queries do not return any entries. Though, without the insider knowledge it's hard to figure out what this means.

Unfortunately my previous setup had to be reinstalled, which means I will need another week to come to the point we see something.

[e-dard]

Just an update: I was able to reproduce this quite trivially with a 1h retention policy and setting the max series to 1 or 2 series. Insert a couple of bits, wait an hour or so and then SHOW SERIES will return nothing (the shard has been dropped) but the limit is still in place and it's not possible to add any new series.

[ahermspacketwerk]

Great, so you don't need my huge data example. If you need anything else, let me know.

[e-dard]

@ahermspacketwerk OK I think I've narrowed this down. I think it's a race inside of the service that manages the labelling of expired shards, and their subsequent removal. If I'm right then I would expect the max series limit to be reset after:

[retention]
    check-interval = "30m0s"

To clarify:

• after check-interval time the retention service checks for expired shards, and marks any as such.
• at the same interval another part of the service should remove them from disk (and therefore the index)
• it's possible due to a race that the second part doesn't complete, and so while it looks like the shards have gone if you use say SHOW SHARDS, they're not actually removed from disk and the index yet.
• wait for check-interval to roll around again, and the shards that were missed will be removed from the index...

I would expect when you experience this issue if you waited another 30m the limit would sort itself out. Hope that makes sense? In the meantime I'm going to put a fix together for this...

[ahermspacketwerk]

As I observed this, the key space was not cleaned up for days. Our software is continuously trying to write data. Even after days there are no entries. "select * ..." returns no values anymore.

If this is only a delayed clean-up, I would expect at least some values.