EPIC: Investigate and fix existing testnet network issues

[evan-forbes]

We're currently observing some oddities around the stability of the mocha testnetwork. While we are producing blocks consistently, we are doing so at variable intervals (anywhere between 8-30s), block producers are producing back to back blocks very frequently, and it is common that peers in the network on different rounds. We need to investigate and fix these issues as soon as possible, and it will likely require further developments for our testing infrastructure #1256 and testnet data collection. It could also be the case that all of the observed issues are caused by the same underlying problem, and that thing problem is blocking us from coming to consensus on the first round.

• Make timeouts dynamic celestia-core#939
• Investigate and fix block producers creating back to back blocks celestia-core#962
• Investigate and fix peers frequently ending up on different rounds of consensus celestia-core#963
• Network Data Collection #1352
• Create data base client to collect arbitrary data remotely celestia-core#968

Note: it's also possible that tendermint's floodsub is simply incapable of consistently producing 8MB blocks in the first round of consensus, which might mean that we are not able to have consistent block times until we improve the gossiping protocol with celestiaorg/celestia-core#884 and celestiaorg/celestia-core#883

[musalbas]

simply incapable of consistently producing 8MB blocks

aren't the vast majority of blocks in mocha much smaller than 8mb though?

[evan-forbes]

aren't the vast majority of blocks in mocha much smaller than 8mb though?

yes, that is correct. I do not believe that this is the reason why we are seeing inconsistent block times, that still requires investigation, and likely changing the timeout commit to be dynamic. I just wanted to note that even after we investigate and implement things like dynamic timeout commits, it is still a possibility that consistently coming to consensus within the first round 95+% of the time for 8MB blocks might not be possible using the existing implementation in a normal network setting.

[evan-forbes]

removing the mainnet milestone as 2MB and 8MB blocks function fine. We still want to fix block prop, but I think that can be separated from this epic