Streaming inmem2tsi conversion. #8963
Conversation
ghost
assigned 
| // Check if shard already has a TSI index. | ||
| indexPath := filepath.Join(path, "index") | ||
| indexPath := filepath.Join(dataDir, "index") |
There was a problem hiding this comment.
This confused me because in your flags section you have "data directory" and "wal directory", which I interpreted as ~/.influxdb/data and ~/.influxdb/wal. I guess by data and wal directories you mean the data and wal database directories?
There was a problem hiding this comment.
I added clarification in 22a0e42.
| } | ||
|
|
||
| switch t := entry.(type) { | ||
| case *tsm1.WriteWALEntry: |
There was a problem hiding this comment.
I think this will create an inconsistent index because it will index everything in the WAL when there could be deletes stored later. Maybe instead, use the CacheLoader to re-create a Cache and iterate over the keys that still remain in the cache?
There was a problem hiding this comment.
I switched it to use the tsm1.CacheLoader instead in 22a0e42. Can you take a look?
benbjohnson
force-pushed
the
streaming-inmem2tsi
branch
from
d4e34da to
759151e
Compare
Changes the `influx_inspect inmem2tsi` tool to stream each TSM/WAL file and convert to a TSI index instead of loading the entire shard's in-memory index first.
benbjohnson
force-pushed
the
streaming-inmem2tsi
branch
from
759151e to
5af2d73
Compare
Overview
Changes the
influx_inspect inmem2tsitool to stream each TSM/WAL file and convert to a TSI index instead of loading the entire shard's in-memory index first.Required for all non-trivial PRs