fix: improved shard deletion #24602
fix: improved shard deletion #24602
Conversation
davidby-influx
changed the title
tsdb/store.go
Outdated
| // TODO(DSB): Should we give up on deleting the shard here, to avoid | ||
| // removing series from the series file that may exist in the shard | ||
| // whose index we could not load? | ||
| } |
There was a problem hiding this comment.
I think the answer is "yes". If we continue deleting, we could lose series. Currently this can only happen during a restore, which probably fixes itself on restore completion, but maybe that changes in the future. Also, if the retention service is running then the shard will eventually be deleted by the retention service, or you will keep getting errors in your log. Since currently the only way to get an error is by running a restore, this will work at some point in the near future.
An alternative is to not remove series if we can't walk all the shards, but then we have phantom series that could impact performance. I prefer we leave the shard on disk until we can properly delete it without losing series or creating a performance issue.
Avoid unnecessarily deleting series from the series file Try harder to delete series from InMem indices Log all errors on shard deletion Closes influxdata#24834
Avoid unnecessarily deleting series from the series file Try harder to delete series from InMem indices Log all errors on shard deletion Closes influxdata#24834
Avoid unnecessarily deleting series from the series file Try harder to delete series from InMem indices Log all errors on shard deletion Closes influxdata#24834
Avoid unnecessarily deleting series from the series file
Try harder to delete series from InMem indices
Log all errors on shard deletion
Closes #24834