Allow Batch Publishing For Applications

[nisdas]

For a more detailed reasoning on the motivation for this:
https://ethresear.ch/t/improving-das-performance-with-gossipsub-batch-publishing/21713

This pull request adds in a new publishing option that allows messages to be published as a batch instead of being queued and sent out individually. Why would this be desired ? Applications can send out multiple messages at the same time. In the event a node has constrained upload bandwidth and these messages are large, the router would take up a longer than desired time sending out D copies to all mesh peers for the first message published. This can add a non-trivial amount of latency for the other messages being propagated to the rest of the network.

This option allows these messages and their copies to be shuffled randomly and then queued in a random order and sent out to their respective peers. This allows for the first copy of any message in the batch to be propagated to the rest of the network much faster. With a first copy being sent out quicker, this leads to earlier message arrival time for the whole network.

• Add Batch Publishing Option.
• Add Batch Message Object to track all the desired message ids and the rpc messages to be published to individual peers.
• Allow Publishing of the whole batch once it is complete.
• Add Tests that the feature works as expected.

[vyzo]

I think the latency would actually increase, as the whole rpc message has tobe received and decoded before any message is processed, no?

[nisdas]

I think the latency would actually increase, as the whole rpc message has tobe received and decoded before any message is processed, no?

Maybe I am missing something, why would that be the case ? This PR just allows you to withhold sending the rpc message to your mesh/direct peers till all the batch message ids have been processed by the gossip router. We then just shuffle the rpcs and send them out so that each message in the batch is sent out 'fairly'

[vyzo]

ah you are not sending as one rpc, so it should be ok.