Enable multithreading on P2P deserialization workload

[asdacap]

Fixes issue with old bodies download capped by unknown force.

Changes:

• Add P2PHandlerThreadCount config.
• Set default thread count to ProcessorCount - 2.

Types of changes

What types of changes does your code introduce?
Put an x in the boxes that apply

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation Update
• Code style update (formatting, renaming)
• Refactoring (no functional changes, no api changes)
• Build related changes
• Other (please describe):

Testing

Requires testing

• Yes
• No

In case you checked yes, did you write tests??

• Yes
• No

Running smoke...

Further comments (optional)

If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution you did and what alternatives you considered, etc...

[MarekM25]

One thing that we should consider. If we do more writes during the old bodies/receipts sync, we will have more problems with attestations during that time. On the other hand, we will end up the sync faster, so not sure what is better.

[asdacap]

We can specify number of thread. Default is core count * 2.

[LukaszRozmej]

Math.Max(Environment.ProcessorCount- 2, 1) ?

[asdacap]

We can also just ask user to set less peer count if sync is too resource intensive. They would have the same problem if they set very high sync peer count anyway.

[asdacap]

3 out of 4 smoke test failed with invalid block. Isolating pr....

[asdacap]

Invalid block is probably due to the package issue. (this was in develop).
Ran again with configurable thread count. Following is old bodies duration.

• Lighthouse: Stuck on slow peer vs 2 hours.
• Lodestar: 1 hours 20 minute vs 1 hours 20 minute.
• Prysm: 1 hours 20 minute vs 1 hours 20 minute.
• Teku: 1 hours 20 minute vs 1 hours 30 minute.

So that is disappointing.

[asdacap]

Mixed with #4725 and #4692 to reduce likelyhood of stuck on slow peer and maximize efficiency.

1 thread vs 8 thread.
Lighthouse: 7 hours 40 minute vs 7 hours.
Lodestar: 5 hours 30 minute vs 5 hours.
Prysm: 5 hours 15 minute vs 4 hours 45 minute.
Teku: 5 hours 45 minute vs 5 houres 15 minute.

Second run have regression across the board, brobably unrelated to this PR.

[asdacap]

Locally is caused segmentation fault even if I set the thread count to 1. Accidentally patched in a known problematic commit.

[asdacap]

Another runs 1thread vs 8thread:

• lighthouse: 7 hours 40 minute vs 7 hours 30 minute.
• lodestar: 5 hours 45 minute vs 5 hours 45 minute.
• prysm: 5 hours 30 minute vs 5 hours 15 minute.
• teku: 5 hours 30 minute vs more than 11 hours. Not complete yet.

[asdacap]

Teku was scheduled on an epyc 7601 while the rest is on epyc 7713. Which explain everything....

[asdacap]

More runs when requested 16 core instance:
1 thread vs 8 thread.

• Lighthouse: 14 hours (schedulet on slow vm) vs 4 hours 35 minute.
• Lodestar; 4 hours 5 minute vs 4 hours.
• Prysm: 4 hours vs 3 hours 50 minute.
• Teku: 3 hours 50 minute vs 4 houres 5 minute.

[asdacap]

Default to CPU count - 2. No regression found after a lot of smoke tests, but usually small improvement exists when measusing whole sync time.

[LukaszRozmej]

Approved, but looks like current improvements are minimal?

[asdacap]

Unfortunately yes.