Gossamer Can't Build Blocks on a Cross-Client v0.9.20 Devnet

[danforbes]

Ensure that Gossamer can author blocks on-par with Substrate nodes in a cross-client devnet.

[kishansagathiya]

I think there are a bunch of issues here. One of them is #2584. This is rather frequent. Like you will rarely not see this issue.

Other issue is that substrate drops gossamer nodes after a while. I don't see a clear pattern over here, like I have gossamer nodes getting dropped after building few blocks and after building 42 blocks as well. I will know more about this problem after I fix #2584. We have seen gossamer dropping issues before and fixed them. This time blocks might be dropping for different reason.

There are few different errors that I see as well regarding epoch handling. But It is not clear how to reproduce them. Some of those error happen on the second run of gossamer node. So, I am guessing based on what ends up getting written to the db (this would mostly depend on where the program is when you stop the gossamer node), and how we get info from db on restarting the node, the errors could change. But again fixing the first issue will give me better idea about other issues. One example of those logs is below

2022-06-06T19:18:10+05:30 EROR failed to persist babe next epoch data: cannot get epoch for block 0 (0xee87cc7ac5e87460094ec4ba29bbf8f28fb7741b69e7edb142233c3dd990b365): header does not contain pre-runtime digest	digest.go:L244	pkg=digest
2022-06-06T19:18:10+05:30 EROR failed to persist babe next epoch config: cannot get epoch for block 0 (0xee87cc7ac5e87460094ec4ba29bbf8f28fb7741b69e7edb142233c3dd990b365): header does not contain pre-runtime digest	digest.go:L249	pkg=digest
2022-06-06T19:18:12+05:30 EROR failed to persist babe next epoch data: cannot get epoch for block 0 (0xee87cc7ac5e87460094ec4ba29bbf8f28fb7741b69e7edb142233c3dd990b365): header does not contain pre-runtime digest	digest.go:L244	pkg=digest
2022-06-06T19:18:12+05:30 EROR failed to persist babe next epoch config: cannot get epoch for block 0 (0xee87cc7ac5e87460094ec4ba29bbf8f28fb7741b69e7edb142233c3dd990b365): header does not contain pre-runtime digest	digest.go:L249	pkg=digest
2022-06-06T19:18:13+05:30 INFO 💤 node waiting, 2 peers connected, head block number 2 with hash 0xf3bffd17c7c882787bfdd303fea4a534fd507e8ca76dac048f728acedc833cc8, finalised block number 0 with hash 0xee87cc7ac5e87460094ec4ba29bbf8f28fb7741b69e7edb142233c3dd990b365	chain_sync.go:L380	pkg=sync
2022-06-06T19:18:14+05:30 EROR failed to persist babe next epoch data: cannot get epoch for block 0 (0xee87cc7ac5e87460094ec4ba29bbf8f28fb7741b69e7edb142233c3dd990b365): header does not contain pre-runtime digest	digest.go:L244	pkg=digest
2022-06-06T19:18:14+05:30 EROR failed to persist babe next epoch config: cannot get epoch for block 0 (0xee87cc7ac5e87460094ec4ba29bbf8f28fb7741b69e7edb142233c3dd990b365): header does not contain pre-runtime digest	digest.go:L249	pkg=digest

[kishansagathiya]

Looks like sometimes blocks built by gossamer are improper with Import failed: Unexpected epoch change

022-06-06 20:44:52 Error with block built on 0x41d30e3127c88687f1ce81a3ffd8adec0a3f59d224987aa6fffd4ba44ecb5d5e: Import failed: Unexpected epoch change    
2022-06-06 20:44:55 💤 Idle (1 peers), best: #3 (0x41d3…5d5e), finalized #0 (0xee87…b365), ⬇ 0.3kiB/s ⬆ 0.2kiB/s    
2022-06-06 20:44:56 🙌 Starting consensus session on top of parent 0x41d30e3127c88687f1ce81a3ffd8adec0a3f59d224987aa6fffd4ba44ecb5d5e    
2022-06-06 20:44:56 🎁 Prepared block for proposing at 4 (0 ms) [hash: 0xd18a26f25d6d3783f6d1b035285ef03261b0fd8f8a5a1908428c67f2f452fbbe; parent_hash: 0x41d3…5d5e; extrinsics (1): [0xd558…c4eb]]    
2022-06-06 20:44:56 🔖 Pre-sealed block for proposal at 4. Hash now 0xc1a41ba127d0c5571cfe9c4f3f271d2c37a4bf4420e7fabdb67cab9e635a721a, previously 0xd18a26f25d6d3783f6d1b035285ef03261b0fd8f8a5a1908428c67f2f452fbbe.    

[kishansagathiya]

Looks like gossamer nodes are getting banned again and that's why they are getting disconnected
https://gist.github.com/kishansagathiya/1c50549051a035806d400a8bbb7b5b96

[kishansagathiya]

Again a different type of error

2022-06-10T11:57:38+05:30 EROR block data processing for block with hash 0xb072c70f001dd09f88c69e0e3dcfe69dffd8f9e6151f69b3120b7be5c909ff9b failed: could not verify block: block producer equivocated	chain_processor.go:L84	pkg=sync

gossamer/lib/babe/verify.go

Lines 367 to 369 in 8582cb2

	if currentBlockProducerIndex == existingBlockProducerIndex && hash != header.Hash() {
	return ErrProducerEquivocated
	}

It basically means same authority is producing two different blocks at the same block number. It must be some bug. Not sure why it would happen otherwise.

[noot]

@kishansagathiya this might happen if there's a fork, and one fork becomes invalidated and the node starts building on another fork and ends up making a block w the same number as previously

I think we might need to remove that check unless there's a way around this