Influxd: Compaction-related panic on HEAD-2016/04/26 #6595
Comments
|
Also the database is corrupted. When restarting |
|
@rw Which config setting was set to From the logs just above that: That looks like a full compaction ran right about the same time as a level 1 compaction did and they both tried to compact the same files into the same destination file causing the panic. I've seen this before, but have not seen it create a TSM file that failed to open. |
|
@jwilder I'll let Robert confirm, but I'm pretty sure this was after writing to the server had stopped. |
|
@toddboom Looks like it's the |
|
@jwilder I set both of those parameters actually :-) To 10s IIRC. Possibly a race condition with the |
|
@jwilder Also, no writes happened during those runs of the database. |
|
@rw You can probably move |
|
I don't know if it's related. I have also compactor issue, and influxdb won't restart: |
|
(replying to comment in # #6595) FWIW, we got this moving from 0.12.1-1 to 0.14.0~n201605160800. Further up (per @jwilder) we have this in the log: Full log at https://gist.github.com/daviesalex/36f3fe30a433dd6a27b2f60cf1598e7f Panic for us: I am testing a downgrade from 0.14.0~n201605160800 to 0.13 stable. Thats starting now, will report back... |
|
@daviesalex Would you be able to send me those two files? |
|
Interestingly, those files do not exist: |
|
Reverting back to 0.13 stable starts, no panic, and shows us all our data. This is a nasty bug in the nightly - do you want a third ticket? |
|
@easyrasta @daviesalex I created a new issue for the interface conversion panic: #6652 The latest nightly reverts the commit that introduced that panic. |
jwilder
added a commit
that referenced
this issue
Jul 22, 2016
Normally, compactions do not conflict on the files they are compacting. If the full cold threshold is set very low, it can cause conflicts where two compactions compact the same files. The full compaction was the only place this could happen as it's planning is greedy. To make this safer for concurrent execution, the compaction tracks which files are current being compacted and prevents any new compactions from starting if the file set overlaps. Fixes #6595
3 tasks
jwilder
added a commit
that referenced
this issue
Jul 26, 2016
Normally, compactions do not conflict on the files they are compacting. If the full cold threshold is set very low, it can cause conflicts where two compactions compact the same files. The full compaction was the only place this could happen as it's planning is greedy. To make this safer for concurrent execution, the compaction tracks which files are current being compacted and prevents any new compactions from starting if the file set overlaps. Fixes #6595
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
During a compaction run (with the compaction intervals in the config file set to
5s),influxdpanic'ed. cc @toddboom @jwilderThe text was updated successfully, but these errors were encountered: