[0.13.0-1] large shards compaction never ends, influx_inspect on the tsm files panics

[t-orb]

System info: [0.13.0-1 on Ubuntu 14.04.4 LTS]

Steps to reproduce:

1. [use default retention policy of each shard covering 1 week]
2. [load in more than a week of data creating big tsm files]
3. [wait until 24hr compaction run starts]

Expected behavior:

If more than 3 TSM files were present I'd expect 1 compaction run across the files

Actual behavior:

It starts compacting and never stops, continues until restarting influxdb.

Additional info: [Include gist of relevant config, logs, etc.]

Here's a peek at one example shard, the logs from last night and influx_inspect failing (until now):

https://gist.github.com/t-orb/bbe0ea12a9bff7bb098728c77d347f23

I asked on Slack and Gunnar Aasen asked me to open an issue.

After listening to one of your webex training sessions I realised that for the amount of data I was putting in the default of each shard covering a week was too big so I switched to 24 hours per shard, all the new shards don't have this issue. So it's likely something going wrong with very large shards/tsm files.

I just realized that running influx_inspect on the files panics.. That's a hint I guess.

The files otherwise appears to work just fine as I can query data in the period just fine.

Thanks for taking a look.

[jwilder]

You need to run influx_inspect dumptsmdev <path>. The dumptsm command is for an older version of TSM files with a different format which is why it panics.

[jwilder]

@t-orb Are you deleting series or measurements by chance? If you delete data that is contained in these TSM files, they will get re-compacted to expunge that data.

[t-orb]

I just updated my gist with a comment containing the dumptsmdev output (no panics).

Never done a delete. No inserts to this database the last day either. It is just running the 24 hour compacting cycle.

The file size of 2148728539 is perhaps a problem?