panic in 'parseKV'

[whyrusleeping]

An ipfs user reported a panic while using badger, occurring here: https://github.com/dgraph-io/badger/blob/master/table/iterator.go#L118

Initial error report (and full stack trace) here: ipfs/kubo#4512 (comment)

[whyrusleeping]

relevant hunk of the stack trace:

panic: runtime error: slice bounds out of range

goroutine 1414 [running]:
gx/ipfs/QmXtSA4dxoDKMzhHkwqeoBoBYASov1ZWnLQo38juKNbe3z/badger/table.(*blockIterator).parseKV(0xc12327ac60, 0x11802ab3ef, 0xc000710003)
        /home/steb/projects/go/src/gx/ipfs/QmXtSA4dxoDKMzhHkwqeoBoBYASov1ZWnLQo38juKNbe3z/badger/table/iterator.go:118 +0x54d
gx/ipfs/QmXtSA4dxoDKMzhHkwqeoBoBYASov1ZWnLQo38juKNbe3z/badger/table.(*blockIterator).Next(0xc12327ac60)
        /home/steb/projects/go/src/gx/ipfs/QmXtSA4dxoDKMzhHkwqeoBoBYASov1ZWnLQo38juKNbe3z/badger/table/iterator.go:154 +0x1ec
gx/ipfs/QmXtSA4dxoDKMzhHkwqeoBoBYASov1ZWnLQo38juKNbe3z/badger/table.(*Iterator).next(0xc0cd84c360)
        /home/steb/projects/go/src/gx/ipfs/QmXtSA4dxoDKMzhHkwqeoBoBYASov1ZWnLQo38juKNbe3z/badger/table/iterator.go:347 +0xc0
gx/ipfs/QmXtSA4dxoDKMzhHkwqeoBoBYASov1ZWnLQo38juKNbe3z/badger/table.(*Iterator).Next(0xc0cd84c360)
        /home/steb/projects/go/src/gx/ipfs/QmXtSA4dxoDKMzhHkwqeoBoBYASov1ZWnLQo38juKNbe3z/badger/table/iterator.go:397 +0x4d
gx/ipfs/QmXtSA4dxoDKMzhHkwqeoBoBYASov1ZWnLQo38juKNbe3z/badger/table.(*ConcatIterator).Next(0xc0d2310a50)
        /home/steb/projects/go/src/gx/ipfs/QmXtSA4dxoDKMzhHkwqeoBoBYASov1ZWnLQo38juKNbe3z/badger/table/iterator.go:507 +0x3a
gx/ipfs/QmXtSA4dxoDKMzhHkwqeoBoBYASov1ZWnLQo38juKNbe3z/badger/y.(*MergeIterator).Next(0xc0d2310be0)
        /home/steb/projects/go/src/gx/ipfs/QmXtSA4dxoDKMzhHkwqeoBoBYASov1ZWnLQo38juKNbe3z/badger/y/iterator.go:216 +0x5b
gx/ipfs/QmXtSA4dxoDKMzhHkwqeoBoBYASov1ZWnLQo38juKNbe3z/badger.(*Iterator).parseItem(0xc0cf33a4d0, 0x1)
        /home/steb/projects/go/src/gx/ipfs/QmXtSA4dxoDKMzhHkwqeoBoBYASov1ZWnLQo38juKNbe3z/badger/iterator.go:388 +0x71c
gx/ipfs/QmXtSA4dxoDKMzhHkwqeoBoBYASov1ZWnLQo38juKNbe3z/badger.(*Iterator).prefetch(0xc0cf33a4d0)
        /home/steb/projects/go/src/gx/ipfs/QmXtSA4dxoDKMzhHkwqeoBoBYASov1ZWnLQo38juKNbe3z/badger/iterator.go:460 +0x9b
gx/ipfs/QmXtSA4dxoDKMzhHkwqeoBoBYASov1ZWnLQo38juKNbe3z/badger.(*Iterator).Seek(0xc0cf33a4d0, 0xc0b0f54c80, 0x42, 0x50)
        /home/steb/projects/go/src/gx/ipfs/QmXtSA4dxoDKMzhHkwqeoBoBYASov1ZWnLQo38juKNbe3z/badger/iterator.go:492 +0x3ee
gx/ipfs/Qmdin8YL17fL1BC5ej6o9b8es6MBoiQjKVdyxEwJh3HVmf/go-ds-badger.(*datastore).QueryNew(0xc042039350, 0xc0b0f54be0, 0x42, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/steb/projects/go/src/gx/ipfs/Qmdin8YL17fL1BC5ej6o9b8es6MBoiQjKVdyxEwJh3HVmf/go-ds-badger/datastore.go:144 +0x1ef
gx/ipfs/Qmdin8YL17fL1BC5ej6o9b8es6MBoiQjKVdyxEwJh3HVmf/go-ds-badger.(*datastore).Query(0xc042039350, 0xc0b0f54be0, 0x42, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/steb/projects/go/src/gx/ipfs/Qmdin8YL17fL1BC5ej6o9b8es6MBoiQjKVdyxEwJh3HVmf/go-ds-badger/datastore.go:133 +0x74
gx/ipfs/QmU7tt6mHJ5Wocjy2omBxpDfN8g9pkRimzJae7EXdrs96k/go-ds-measure.(*measure).Query(0xc04213c5a0, 0xc0b0f54be0, 0x42, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/steb/projects/go/src/gx/ipfs/QmU7tt6mHJ5Wocjy2omBxpDfN8g9pkRimzJae7EXdrs96k/go-ds-measure/measure.go:141 +0x12e
gx/ipfs/QmU7tt6mHJ5Wocjy2omBxpDfN8g9pkRimzJae7EXdrs96k/go-ds-measure.(*measure).Query(0xc04213c6c0, 0xc0b0f54be0, 0x42, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/steb/projects/go/src/gx/ipfs/QmU7tt6mHJ5Wocjy2omBxpDfN8g9pkRimzJae7EXdrs96k/go-ds-measure/measure.go:141 +0x12e

[janardhan1993]

@whyrusleeping Which version of badger are you using ? Are you using master?
What's the typical size of your keys ?

[whyrusleeping]

@janardhan1993 i'll have to check the exact commit we vendored from, but you can find the code we're using here: https://ipfs.io/ipfs/QmXtSA4dxoDKMzhHkwqeoBoBYASov1ZWnLQo38juKNbe3z/badger

Our keys are generally around 40 bytes long if i remember correctly.

cc @magik6k who has been working closely with badger for us.

[janardhan1993]

@whyrusleeping Can you also share the badger directory if possible, it would help us in debugging faster.

[whyrusleeping]

@janardhan1993 I don't know if we can. Its from a user who reported the issue. I'll ask.

[janardhan1993]

@whyrusleeping It would be great if you can share a way to reproduce this.

[whyrusleeping]

@janardhan1993 I'm working on it, the user was just running ipfs using badger on windows.

We're using commit a9ee745 of badger. We're suspecting that its either a race condition or a memory corruption issue somewhere (theres a good amount of 'unsafe' usage in badger that we havent looked through in the skiplist)

[manishrjain]

Looks like this wasn't an issue with Badger. Closing.

ipfs/kubo#4512 (comment)