Perf/improve snap request concurrency

[asdacap]

• Improve snap request concurrency by splitting top level account range into n configurable, fixed sized partition.
• Improve my snap sync time by about 20% to 25%, but more importantly, reliably saturate my ssd, so snap sync now reliably take less than one hour to complete.
• This works as currently only some peers are used in snap sync as it is limited by the top level account range progress to know what else to request. If the network is fast, and nethermind got lucky to get a fast peer (and it dispatch account range request to that peer), the sync will be quick. Otherwise, it does not matter how fast your internet and ssd are.
• This change split the top level account range into n (8 by default) partition and dispatch them concurrently to multiple peer, significantly increase the request rate.
• Snap protocol already have limit parameters, so this works well.

Changes

• Add config to specify num of partition.
• Dispatch multiple concurrent account range request where possible.

Types of changes

What types of changes does your code introduce?

• Optimization

Testing

Requires testing

• Yes
• No

If yes, did you write tests?

• Yes
• No

Notes on testing

• Multiple successful mainnet sync.
• Had to stop mainnet VerifyTrie, its taking too much time. Last progress: 2023-02-28 00:03:43.0144|WARN|13|Collected info from 1100001490 nodes. Missing CODE 0 STATE 0 STORAGE 0 .
• Goerli can sync
• Sepolia still looking for peer (known issue) and my new ISP have this nasty habit of not working randomly when I start nethermind. It's synced now. Don't know when, just left it on for the whole day.

Other notes

• It seems to be possible to remove the commit lock by creating a new triestore every time. This further reduce sync time by another 20%. I did not include that here as that seems like something that will silently break things, maybe on a different PR.

[dceleda]

Now there will be always healing required, even for small states because stitching between partitions won't work but I don't think it's a problem.
In general looks good. A few minor comments added.
@kamilchodola we need to run multiple test syncs on mainnet and then Verify the whole tree although it's going to be slow.
We should also test scenarios when we close the node after Phase 1 and start again etc.

[dceleda]

@damian-orzechowski please also review.

[asdacap]

Aside from the fact that the partitions is going to cause interleaving range from different root hash, why does the stitching does not work? Or do you mean it does not work as well?

[dceleda]

Aside from the fact that the partitions is going to cause interleaving range from different root hash, why does the stitching does not work? Or do you mean it does not work as well?

I'll explain on a call.