Investigate long epoch transition during Pyrmont sync

[twoeths]

I was able to catch a 4.84s epoch transition in my local environment using Chrome profiler

An overview:

• prepareEpochProcessState takes 530ms ++, and there are some small executions happened after that
• processRewardsAndPenalties takes 2.4s, getAttestationDeltas takes 1.17s, epochBalances setter takes 764ms
• rotateEpoch takes more than 1s, computeEpochShuffling takes 687ms
• hashTreeRoot takes 425ms

event loop lag somehow was at 1.5s only, I'm attaching the snapshot as well.
CPU-20210215T150254-4s-epoch-transition.cpuprofile.zip

[wemeetagain]

IMO we have likely milked as much performance out of things as reasonable and should just attempt to mitigate the slowness.
We should just break up epoch processing with await sleep(0); to just prevent any single piece of the epoch transition from taking

For instance, there's not much that can be done regarding re-hashing the balances. It's just computationally intensive work that has to be done.
Similar issue with just setting the state balances. The balances list needs to be converted to a tree, and that just takes time.

If we yield back to the event loop in between our longer-running subtasks, it likely won't even matter that the epoch transition takes time.

[dapplion]

@wemeetagain I had a chat with Paul from Lighthouse and their epoch transitions take 20-30 ms, 100 times faster than ours. That's an insane difference that should be at least investigated.

[wemeetagain]

It looks like we might not be iterating thru our cached validators/balances where we can?

[twoeths]

getAttesationDeltas()/BigInt performance is really unpredictable (ranging from 98ms to 1375ms) as shown in PR #2064 , one possibility is to implement the function in assemblyscript. From a quick look, we can just pass balances, config params (5 of them), EpochProcess (may pass 20 different params), finalized epoch and receive back new balances (if assemblyscript does not support passing objects as argument).

not sure if we want to do a POC for it at this time.

[dapplion]

not sure if we want to do a POC for it at this time.

I think it's worth it! Using assembly script won't introduce new complexities such as multi-treading

[twoeths]

so I tried assemblyscript, one issue I had was OOM issue although I configured different gc strategies there and manually called __collect() (tried with v0.9.2), so for now I think this is more suitable to functions where we can allocate a static memory allocation similar to how as-sha256 was implemented.
Also in order to pass bigint to wasm we have to use --experimental-wasm-bigint flag which is a different barrier, not sure when that become official in nodejs.

[twoeths]

current epoch transition for Pyrmont at around slot 363972 is 890ms.

@ChainSafe/eth2-0 I think the last thing to do for this issue is to add await sleep(0) in state transition and make sure we don't block for 200ms?

[dapplion]

IMO we have likely milked as much performance out of things as reasonable and should just attempt to mitigate the slowness.
We should just break up epoch processing with await sleep(0); to just prevent any single piece of the epoch transition from taking

For instance, there's not much that can be done regarding re-hashing the balances. It's just computationally intensive work that has to be done.
Similar issue with just setting the state balances. The balances list needs to be converted to a tree, and that just takes time.

If we yield back to the event loop in between our longer-running subtasks, it likely won't even matter that the epoch transition takes time.

Oh man, we have milked so much more performance out of this 🎉

Closing, add new issues for specific improvement strategies