Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save and use full LightClientUpdate instead of partial #4702

Merged
merged 3 commits into from
Oct 30, 2022

Conversation

g11tech
Copy link
Contributor

@g11tech g11tech commented Oct 29, 2022

In the previous scheme of things, the sync committee branch corresponding to finalized checkpoint's state was fetched and bundled, but that shifted from finalized checkpoints state to the attested header's state.

But the non checkpoint syncComitteeWitness data gets pruned later and hence the best partial update was no more servicable causing broken get update api.

This PR fixes by storing the full update instead of partial because all corresponding data is available while storing the update, so that the data availability doesn't become the issue post pruning.

Console log of the light client demo which can now properly process the lightclient updates to sync to the current (the new url fetch errors are because of the latest merged p2p PR which might have updated some routes, the light client demo to be independently updated)

image

@g11tech g11tech requested a review from a team as a code owner October 29, 2022 14:58
@github-actions
Copy link
Contributor

github-actions bot commented Oct 29, 2022

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 132b4fd Previous: 03034ff Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 2.3962 ms/op 2.6633 ms/op 0.90
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 87.356 us/op 88.729 us/op 0.98
BLS verify - blst-native 2.3384 ms/op 2.0315 ms/op 1.15
BLS verifyMultipleSignatures 3 - blst-native 4.7986 ms/op 4.2187 ms/op 1.14
BLS verifyMultipleSignatures 8 - blst-native 10.308 ms/op 8.8473 ms/op 1.17
BLS verifyMultipleSignatures 32 - blst-native 37.032 ms/op 32.038 ms/op 1.16
BLS aggregatePubkeys 32 - blst-native 50.444 us/op 45.464 us/op 1.11
BLS aggregatePubkeys 128 - blst-native 194.85 us/op 173.25 us/op 1.12
getAttestationsForBlock 107.98 ms/op 104.90 ms/op 1.03
isKnown best case - 1 super set check 532.00 ns/op 475.00 ns/op 1.12
isKnown normal case - 2 super set checks 507.00 ns/op 448.00 ns/op 1.13
isKnown worse case - 16 super set checks 524.00 ns/op 454.00 ns/op 1.15
CheckpointStateCache - add get delete 11.192 us/op 10.342 us/op 1.08
validate gossip signedAggregateAndProof - struct 5.4804 ms/op 4.7645 ms/op 1.15
validate gossip attestation - struct 2.5094 ms/op 2.2500 ms/op 1.12
pickEth1Vote - no votes 2.5988 ms/op 2.2044 ms/op 1.18
pickEth1Vote - max votes 21.992 ms/op 23.368 ms/op 0.94
pickEth1Vote - Eth1Data hashTreeRoot value x2048 13.568 ms/op 13.354 ms/op 1.02
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 23.467 ms/op 23.282 ms/op 1.01
pickEth1Vote - Eth1Data fastSerialize value x2048 1.7615 ms/op 1.6636 ms/op 1.06
pickEth1Vote - Eth1Data fastSerialize tree x2048 15.221 ms/op 14.958 ms/op 1.02
bytes32 toHexString 1.2480 us/op 1.1340 us/op 1.10
bytes32 Buffer.toString(hex) 855.00 ns/op 749.00 ns/op 1.14
bytes32 Buffer.toString(hex) from Uint8Array 1.1200 us/op 1.0480 us/op 1.07
bytes32 Buffer.toString(hex) + 0x 837.00 ns/op 761.00 ns/op 1.10
Object access 1 prop 0.40000 ns/op 0.37300 ns/op 1.07
Map access 1 prop 0.35600 ns/op 0.33400 ns/op 1.07
Object get x1000 16.279 ns/op 15.228 ns/op 1.07
Map get x1000 1.0680 ns/op 0.93200 ns/op 1.15
Object set x1000 113.29 ns/op 115.42 ns/op 0.98
Map set x1000 79.020 ns/op 79.539 ns/op 0.99
Return object 10000 times 0.43730 ns/op 0.39280 ns/op 1.11
Throw Error 10000 times 8.6685 us/op 7.5995 us/op 1.14
fastMsgIdFn sha256 / 200 bytes 5.7120 us/op 4.8810 us/op 1.17
fastMsgIdFn h32 xxhash / 200 bytes 623.00 ns/op 584.00 ns/op 1.07
fastMsgIdFn h64 xxhash / 200 bytes 790.00 ns/op 847.00 ns/op 0.93
fastMsgIdFn sha256 / 1000 bytes 16.968 us/op 15.259 us/op 1.11
fastMsgIdFn h32 xxhash / 1000 bytes 811.00 ns/op 764.00 ns/op 1.06
fastMsgIdFn h64 xxhash / 1000 bytes 883.00 ns/op 986.00 ns/op 0.90
fastMsgIdFn sha256 / 10000 bytes 143.37 us/op 132.54 us/op 1.08
fastMsgIdFn h32 xxhash / 10000 bytes 2.7780 us/op 2.6330 us/op 1.06
fastMsgIdFn h64 xxhash / 10000 bytes 2.0970 us/op 1.9540 us/op 1.07
enrSubnets - fastDeserialize 64 bits 3.1160 us/op 3.1120 us/op 1.00
enrSubnets - ssz BitVector 64 bits 852.00 ns/op 825.00 ns/op 1.03
enrSubnets - fastDeserialize 4 bits 453.00 ns/op 462.00 ns/op 0.98
enrSubnets - ssz BitVector 4 bits 866.00 ns/op 856.00 ns/op 1.01
prioritizePeers score -10:0 att 32-0.1 sync 2-0 114.30 us/op 109.94 us/op 1.04
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 157.51 us/op 138.05 us/op 1.14
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 277.19 us/op 282.16 us/op 0.98
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 613.35 us/op 579.39 us/op 1.06
prioritizePeers score 0:0 att 64-1 sync 4-1 588.51 us/op 557.59 us/op 1.06
RateTracker 1000000 limit, 1 obj count per request 212.78 ns/op 195.17 ns/op 1.09
RateTracker 1000000 limit, 2 obj count per request 157.09 ns/op 145.82 ns/op 1.08
RateTracker 1000000 limit, 4 obj count per request 127.69 ns/op 121.97 ns/op 1.05
RateTracker 1000000 limit, 8 obj count per request 115.04 ns/op 113.06 ns/op 1.02
RateTracker with prune 5.1480 us/op 5.5340 us/op 0.93
array of 16000 items push then shift 5.2805 us/op 4.9256 us/op 1.07
LinkedList of 16000 items push then shift 18.795 ns/op 20.702 ns/op 0.91
array of 16000 items push then pop 260.13 ns/op 252.38 ns/op 1.03
LinkedList of 16000 items push then pop 17.194 ns/op 19.088 ns/op 0.90
array of 24000 items push then shift 8.0143 us/op 7.4436 us/op 1.08
LinkedList of 24000 items push then shift 19.910 ns/op 21.514 ns/op 0.93
array of 24000 items push then pop 268.07 ns/op 218.80 ns/op 1.23
LinkedList of 24000 items push then pop 18.635 ns/op 18.841 ns/op 0.99
intersect bitArray bitLen 8 12.678 ns/op 11.986 ns/op 1.06
intersect array and set length 8 198.64 ns/op 207.38 ns/op 0.96
intersect bitArray bitLen 128 75.695 ns/op 67.573 ns/op 1.12
intersect array and set length 128 2.4958 us/op 2.4040 us/op 1.04
Buffer.concat 32 items 3.0000 ns/op 2.2660 ns/op 1.32
pass gossip attestations to forkchoice per slot 4.8374 ms/op 4.2201 ms/op 1.15
computeDeltas 5.6350 ms/op 5.2885 ms/op 1.07
computeProposerBoostScoreFromBalances 914.65 us/op 830.59 us/op 1.10
altair processAttestation - 250000 vs - 7PWei normalcase 4.6082 ms/op 5.5975 ms/op 0.82
altair processAttestation - 250000 vs - 7PWei worstcase 7.5709 ms/op 8.4704 ms/op 0.89
altair processAttestation - setStatus - 1/6 committees join 275.93 us/op 229.54 us/op 1.20
altair processAttestation - setStatus - 1/3 committees join 515.03 us/op 481.02 us/op 1.07
altair processAttestation - setStatus - 1/2 committees join 724.07 us/op 648.91 us/op 1.12
altair processAttestation - setStatus - 2/3 committees join 965.50 us/op 867.19 us/op 1.11
altair processAttestation - setStatus - 4/5 committees join 1.3100 ms/op 1.2517 ms/op 1.05
altair processAttestation - setStatus - 100% committees join 1.5391 ms/op 1.4553 ms/op 1.06
altair processBlock - 250000 vs - 7PWei normalcase 32.668 ms/op 31.807 ms/op 1.03
altair processBlock - 250000 vs - 7PWei normalcase hashState 46.095 ms/op 41.229 ms/op 1.12
altair processBlock - 250000 vs - 7PWei worstcase 101.79 ms/op 104.57 ms/op 0.97
altair processBlock - 250000 vs - 7PWei worstcase hashState 119.94 ms/op 113.20 ms/op 1.06
phase0 processBlock - 250000 vs - 7PWei normalcase 4.3713 ms/op 4.3620 ms/op 1.00
phase0 processBlock - 250000 vs - 7PWei worstcase 57.305 ms/op 52.956 ms/op 1.08
altair processEth1Data - 250000 vs - 7PWei normalcase 810.94 us/op 1.0078 ms/op 0.80
Tree 40 250000 create 829.21 ms/op 974.87 ms/op 0.85
Tree 40 250000 get(125000) 332.82 ns/op 304.58 ns/op 1.09
Tree 40 250000 set(125000) 2.8603 us/op 3.2776 us/op 0.87
Tree 40 250000 toArray() 43.473 ms/op 35.079 ms/op 1.24
Tree 40 250000 iterate all - toArray() + loop 44.447 ms/op 36.582 ms/op 1.21
Tree 40 250000 iterate all - get(i) 140.48 ms/op 124.32 ms/op 1.13
MutableVector 250000 create 17.815 ms/op 20.347 ms/op 0.88
MutableVector 250000 get(125000) 14.668 ns/op 12.875 ns/op 1.14
MutableVector 250000 set(125000) 751.46 ns/op 866.23 ns/op 0.87
MutableVector 250000 toArray() 7.8045 ms/op 7.8247 ms/op 1.00
MutableVector 250000 iterate all - toArray() + loop 7.6814 ms/op 7.8867 ms/op 0.97
MutableVector 250000 iterate all - get(i) 3.4260 ms/op 3.4756 ms/op 0.99
Array 250000 create 6.9758 ms/op 8.4728 ms/op 0.82
Array 250000 clone - spread 3.8328 ms/op 4.7555 ms/op 0.81
Array 250000 get(125000) 1.6230 ns/op 2.0140 ns/op 0.81
Array 250000 set(125000) 1.6060 ns/op 1.9500 ns/op 0.82
Array 250000 iterate all - loop 150.10 us/op 165.05 us/op 0.91
effectiveBalanceIncrements clone Uint8Array 300000 111.70 us/op 132.98 us/op 0.84
effectiveBalanceIncrements clone MutableVector 300000 747.00 ns/op 1.4770 us/op 0.51
effectiveBalanceIncrements rw all Uint8Array 300000 303.82 us/op 318.93 us/op 0.95
effectiveBalanceIncrements rw all MutableVector 300000 205.53 ms/op 326.42 ms/op 0.63
phase0 afterProcessEpoch - 250000 vs - 7PWei 227.67 ms/op 210.43 ms/op 1.08
phase0 beforeProcessEpoch - 250000 vs - 7PWei 79.201 ms/op 131.01 ms/op 0.60
altair processEpoch - mainnet_e81889 702.63 ms/op 717.42 ms/op 0.98
mainnet_e81889 - altair beforeProcessEpoch 110.89 ms/op 117.03 ms/op 0.95
mainnet_e81889 - altair processJustificationAndFinalization 67.566 us/op 79.490 us/op 0.85
mainnet_e81889 - altair processInactivityUpdates 12.204 ms/op 12.572 ms/op 0.97
mainnet_e81889 - altair processRewardsAndPenalties 185.91 ms/op 200.02 ms/op 0.93
mainnet_e81889 - altair processRegistryUpdates 16.237 us/op 13.819 us/op 1.17
mainnet_e81889 - altair processSlashings 2.9810 us/op 4.0540 us/op 0.74
mainnet_e81889 - altair processEth1DataReset 3.4830 us/op 4.2790 us/op 0.81
mainnet_e81889 - altair processEffectiveBalanceUpdates 3.4176 ms/op 2.9820 ms/op 1.15
mainnet_e81889 - altair processSlashingsReset 29.060 us/op 21.448 us/op 1.35
mainnet_e81889 - altair processRandaoMixesReset 25.276 us/op 22.909 us/op 1.10
mainnet_e81889 - altair processHistoricalRootsUpdate 3.9120 us/op 3.5440 us/op 1.10
mainnet_e81889 - altair processParticipationFlagUpdates 15.514 us/op 15.137 us/op 1.02
mainnet_e81889 - altair processSyncCommitteeUpdates 3.5160 us/op 3.2410 us/op 1.08
mainnet_e81889 - altair afterProcessEpoch 213.14 ms/op 191.54 ms/op 1.11
phase0 processEpoch - mainnet_e58758 622.44 ms/op 687.64 ms/op 0.91
mainnet_e58758 - phase0 beforeProcessEpoch 312.49 ms/op 245.94 ms/op 1.27
mainnet_e58758 - phase0 processJustificationAndFinalization 62.741 us/op 59.405 us/op 1.06
mainnet_e58758 - phase0 processRewardsAndPenalties 91.715 ms/op 144.78 ms/op 0.63
mainnet_e58758 - phase0 processRegistryUpdates 34.115 us/op 34.610 us/op 0.99
mainnet_e58758 - phase0 processSlashings 3.3190 us/op 3.6850 us/op 0.90
mainnet_e58758 - phase0 processEth1DataReset 3.5850 us/op 3.8630 us/op 0.93
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 2.0943 ms/op 2.2820 ms/op 0.92
mainnet_e58758 - phase0 processSlashingsReset 17.134 us/op 18.009 us/op 0.95
mainnet_e58758 - phase0 processRandaoMixesReset 27.029 us/op 22.324 us/op 1.21
mainnet_e58758 - phase0 processHistoricalRootsUpdate 4.4050 us/op 3.7900 us/op 1.16
mainnet_e58758 - phase0 processParticipationRecordUpdates 25.468 us/op 24.303 us/op 1.05
mainnet_e58758 - phase0 afterProcessEpoch 181.20 ms/op 162.69 ms/op 1.11
phase0 processEffectiveBalanceUpdates - 250000 normalcase 2.5386 ms/op 2.2726 ms/op 1.12
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 3.4208 ms/op 2.4939 ms/op 1.37
altair processInactivityUpdates - 250000 normalcase 56.458 ms/op 49.815 ms/op 1.13
altair processInactivityUpdates - 250000 worstcase 53.124 ms/op 61.738 ms/op 0.86
phase0 processRegistryUpdates - 250000 normalcase 29.281 us/op 26.639 us/op 1.10
phase0 processRegistryUpdates - 250000 badcase_full_deposits 560.63 us/op 501.43 us/op 1.12
phase0 processRegistryUpdates - 250000 worstcase 0.5 263.85 ms/op 250.56 ms/op 1.05
altair processRewardsAndPenalties - 250000 normalcase 161.48 ms/op 131.30 ms/op 1.23
altair processRewardsAndPenalties - 250000 worstcase 130.46 ms/op 94.644 ms/op 1.38
phase0 getAttestationDeltas - 250000 normalcase 14.865 ms/op 14.054 ms/op 1.06
phase0 getAttestationDeltas - 250000 worstcase 14.138 ms/op 14.453 ms/op 0.98
phase0 processSlashings - 250000 worstcase 6.5964 ms/op 6.2150 ms/op 1.06
altair processSyncCommitteeUpdates - 250000 357.74 ms/op 325.45 ms/op 1.10
BeaconState.hashTreeRoot - No change 642.00 ns/op 620.00 ns/op 1.04
BeaconState.hashTreeRoot - 1 full validator 79.636 us/op 74.083 us/op 1.07
BeaconState.hashTreeRoot - 32 full validator 867.49 us/op 637.72 us/op 1.36
BeaconState.hashTreeRoot - 512 full validator 8.2633 ms/op 7.7035 ms/op 1.07
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 96.695 us/op 101.86 us/op 0.95
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.5124 ms/op 1.6778 ms/op 0.90
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 18.547 ms/op 19.005 ms/op 0.98
BeaconState.hashTreeRoot - 1 balances 73.514 us/op 69.878 us/op 1.05
BeaconState.hashTreeRoot - 32 balances 674.26 us/op 749.93 us/op 0.90
BeaconState.hashTreeRoot - 512 balances 7.1112 ms/op 6.9126 ms/op 1.03
BeaconState.hashTreeRoot - 250000 balances 128.33 ms/op 101.51 ms/op 1.26
aggregationBits - 2048 els - zipIndexesInBitList 29.316 us/op 42.430 us/op 0.69
regular array get 100000 times 58.765 us/op 51.524 us/op 1.14
wrappedArray get 100000 times 59.177 us/op 52.621 us/op 1.12
arrayWithProxy get 100000 times 36.754 ms/op 32.295 ms/op 1.14
ssz.Root.equals 544.00 ns/op 598.00 ns/op 0.91
byteArrayEquals 523.00 ns/op 583.00 ns/op 0.90
shuffle list - 16384 els 12.140 ms/op 11.105 ms/op 1.09
shuffle list - 250000 els 179.22 ms/op 157.01 ms/op 1.14
processSlot - 1 slots 17.426 us/op 16.067 us/op 1.08
processSlot - 32 slots 2.3063 ms/op 2.0942 ms/op 1.10
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 472.64 us/op 437.79 us/op 1.08
getCommitteeAssignments - req 1 vs - 250000 vc 5.5819 ms/op 6.1717 ms/op 0.90
getCommitteeAssignments - req 100 vs - 250000 vc 8.1665 ms/op 8.5309 ms/op 0.96
getCommitteeAssignments - req 1000 vs - 250000 vc 8.7771 ms/op 9.0009 ms/op 0.98
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 10.410 ns/op 11.560 ns/op 0.90
state getBlockRootAtSlot - 250000 vs - 7PWei 1.2801 us/op 1.5781 us/op 0.81
computeProposers - vc 250000 21.044 ms/op 22.499 ms/op 0.94
computeEpochShuffling - vc 250000 185.59 ms/op 190.67 ms/op 0.97
getNextSyncCommittee - vc 250000 362.03 ms/op 388.35 ms/op 0.93

by benchmarkbot/action

@@ -60,7 +60,7 @@ export enum Bucket {
lightClient_syncCommittee = 52, // Root(altair.SyncCommittee) -> altair.SyncCommittee
// TODO: May be redundant to block stores
lightClient_checkpointHeader = 53, // BlockRoot -> phase0.BeaconBlockHeader
lightClient_bestPartialLightClientUpdate = 54, // SyncPeriod -> PartialLightClientUpdate
lightClient_bestLightClientUpdate = 54, // SyncPeriod -> LightClientUpdate
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we use a new bucket for a non-errored migration?

@g11tech g11tech changed the title Save full LightClientUpdate instead of partial Save and use full LightClientUpdate instead of partial Oct 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants