Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve fork_choice spec test runner's time #5277

Merged
merged 1 commit into from
Mar 18, 2023
Merged

Improve fork_choice spec test runner's time #5277

merged 1 commit into from
Mar 18, 2023

Conversation

twoeths
Copy link
Contributor

@twoeths twoeths commented Mar 17, 2023

Motivation

  • Fork choice spec test runs really slow (20x compared to v1.5.0) due to loading ckzg for all forks

Description

  • Only load ckzg for deneb in fork_choice spec test runner
  • Fix isForkWithdrawals util

@twoeths twoeths requested a review from a team as a code owner March 17, 2023 04:27
@github-actions
Copy link
Contributor

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: bd92ecf Previous: 3d937ee Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 605.36 us/op 916.25 us/op 0.66
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 59.367 us/op 53.810 us/op 1.10
BLS verify - blst-native 1.2499 ms/op 1.2654 ms/op 0.99
BLS verifyMultipleSignatures 3 - blst-native 2.5884 ms/op 2.5854 ms/op 1.00
BLS verifyMultipleSignatures 8 - blst-native 5.8869 ms/op 5.5368 ms/op 1.06
BLS verifyMultipleSignatures 32 - blst-native 20.018 ms/op 20.005 ms/op 1.00
BLS aggregatePubkeys 32 - blst-native 26.511 us/op 26.915 us/op 0.98
BLS aggregatePubkeys 128 - blst-native 103.21 us/op 105.80 us/op 0.98
getAttestationsForBlock 62.524 ms/op 60.640 ms/op 1.03
isKnown best case - 1 super set check 289.00 ns/op 270.00 ns/op 1.07
isKnown normal case - 2 super set checks 281.00 ns/op 261.00 ns/op 1.08
isKnown worse case - 16 super set checks 280.00 ns/op 256.00 ns/op 1.09
CheckpointStateCache - add get delete 5.6290 us/op 5.7750 us/op 0.97
validate gossip signedAggregateAndProof - struct 2.8647 ms/op 2.8957 ms/op 0.99
validate gossip attestation - struct 1.3702 ms/op 1.3845 ms/op 0.99
pickEth1Vote - no votes 1.4181 ms/op 1.3991 ms/op 1.01
pickEth1Vote - max votes 10.860 ms/op 11.307 ms/op 0.96
pickEth1Vote - Eth1Data hashTreeRoot value x2048 9.9151 ms/op 10.542 ms/op 0.94
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 16.419 ms/op 17.200 ms/op 0.95
pickEth1Vote - Eth1Data fastSerialize value x2048 803.05 us/op 875.70 us/op 0.92
pickEth1Vote - Eth1Data fastSerialize tree x2048 5.8149 ms/op 10.834 ms/op 0.54
bytes32 toHexString 592.00 ns/op 652.00 ns/op 0.91
bytes32 Buffer.toString(hex) 406.00 ns/op 498.00 ns/op 0.82
bytes32 Buffer.toString(hex) from Uint8Array 637.00 ns/op 689.00 ns/op 0.92
bytes32 Buffer.toString(hex) + 0x 444.00 ns/op 481.00 ns/op 0.92
Object access 1 prop 0.20100 ns/op 0.23100 ns/op 0.87
Map access 1 prop 0.16500 ns/op 0.18300 ns/op 0.90
Object get x1000 6.9850 ns/op 7.8700 ns/op 0.89
Map get x1000 0.68100 ns/op 0.69400 ns/op 0.98
Object set x1000 79.906 ns/op 88.857 ns/op 0.90
Map set x1000 57.542 ns/op 62.176 ns/op 0.93
Return object 10000 times 0.26130 ns/op 0.37580 ns/op 0.70
Throw Error 10000 times 4.8127 us/op 4.7290 us/op 1.02
fastMsgIdFn sha256 / 200 bytes 3.9280 us/op 3.8360 us/op 1.02
fastMsgIdFn h32 xxhash / 200 bytes 359.00 ns/op 343.00 ns/op 1.05
fastMsgIdFn h64 xxhash / 200 bytes 545.00 ns/op 532.00 ns/op 1.02
fastMsgIdFn sha256 / 1000 bytes 12.317 us/op 12.726 us/op 0.97
fastMsgIdFn h32 xxhash / 1000 bytes 463.00 ns/op 498.00 ns/op 0.93
fastMsgIdFn h64 xxhash / 1000 bytes 534.00 ns/op 603.00 ns/op 0.89
fastMsgIdFn sha256 / 10000 bytes 108.01 us/op 111.18 us/op 0.97
fastMsgIdFn h32 xxhash / 10000 bytes 2.0300 us/op 2.1370 us/op 0.95
fastMsgIdFn h64 xxhash / 10000 bytes 1.4800 us/op 1.5970 us/op 0.93
enrSubnets - fastDeserialize 64 bits 1.6190 us/op 1.9970 us/op 0.81
enrSubnets - ssz BitVector 64 bits 622.00 ns/op 690.00 ns/op 0.90
enrSubnets - fastDeserialize 4 bits 209.00 ns/op 235.00 ns/op 0.89
enrSubnets - ssz BitVector 4 bits 638.00 ns/op 686.00 ns/op 0.93
prioritizePeers score -10:0 att 32-0.1 sync 2-0 131.40 us/op 118.59 us/op 1.11
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 172.25 us/op 184.88 us/op 0.93
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 232.88 us/op 220.62 us/op 1.06
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 370.84 us/op 413.17 us/op 0.90
prioritizePeers score 0:0 att 64-1 sync 4-1 435.98 us/op 557.44 us/op 0.78
array of 16000 items push then shift 1.8112 us/op 1.8720 us/op 0.97
LinkedList of 16000 items push then shift 11.063 ns/op 12.662 ns/op 0.87
array of 16000 items push then pop 114.94 ns/op 128.98 ns/op 0.89
LinkedList of 16000 items push then pop 10.124 ns/op 11.869 ns/op 0.85
array of 24000 items push then shift 2.5011 us/op 2.5959 us/op 0.96
LinkedList of 24000 items push then shift 10.580 ns/op 11.948 ns/op 0.89
array of 24000 items push then pop 103.85 ns/op 87.888 ns/op 1.18
LinkedList of 24000 items push then pop 9.6230 ns/op 8.9910 ns/op 1.07
intersect bitArray bitLen 8 13.915 ns/op 13.683 ns/op 1.02
intersect array and set length 8 90.836 ns/op 98.081 ns/op 0.93
intersect bitArray bitLen 128 47.574 ns/op 47.347 ns/op 1.00
intersect array and set length 128 1.2154 us/op 1.3322 us/op 0.91
Buffer.concat 32 items 2.8430 us/op 2.8680 us/op 0.99
Uint8Array.set 32 items 2.4890 us/op 2.7580 us/op 0.90
pass gossip attestations to forkchoice per slot 3.9536 ms/op 4.3230 ms/op 0.91
computeDeltas 3.2920 ms/op 4.0910 ms/op 0.80
computeProposerBoostScoreFromBalances 1.8469 ms/op 1.9507 ms/op 0.95
altair processAttestation - 250000 vs - 7PWei normalcase 3.0374 ms/op 2.9759 ms/op 1.02
altair processAttestation - 250000 vs - 7PWei worstcase 3.8161 ms/op 3.8141 ms/op 1.00
altair processAttestation - setStatus - 1/6 committees join 148.14 us/op 144.65 us/op 1.02
altair processAttestation - setStatus - 1/3 committees join 289.19 us/op 283.26 us/op 1.02
altair processAttestation - setStatus - 1/2 committees join 368.46 us/op 383.12 us/op 0.96
altair processAttestation - setStatus - 2/3 committees join 459.49 us/op 477.50 us/op 0.96
altair processAttestation - setStatus - 4/5 committees join 649.51 us/op 673.24 us/op 0.96
altair processAttestation - setStatus - 100% committees join 759.03 us/op 781.89 us/op 0.97
altair processBlock - 250000 vs - 7PWei normalcase 19.181 ms/op 20.176 ms/op 0.95
altair processBlock - 250000 vs - 7PWei normalcase hashState 25.105 ms/op 28.264 ms/op 0.89
altair processBlock - 250000 vs - 7PWei worstcase 47.837 ms/op 50.613 ms/op 0.95
altair processBlock - 250000 vs - 7PWei worstcase hashState 74.384 ms/op 69.434 ms/op 1.07
phase0 processBlock - 250000 vs - 7PWei normalcase 2.0311 ms/op 2.2903 ms/op 0.89
phase0 processBlock - 250000 vs - 7PWei worstcase 29.531 ms/op 30.797 ms/op 0.96
altair processEth1Data - 250000 vs - 7PWei normalcase 490.35 us/op 519.50 us/op 0.94
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 8.8220 us/op 8.4790 us/op 1.04
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 27.438 us/op 28.113 us/op 0.98
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 12.051 us/op 10.836 us/op 1.11
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 8.4910 us/op 8.8980 us/op 0.95
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 98.457 us/op 111.10 us/op 0.89
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 656.42 us/op 692.00 us/op 0.95
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 918.65 us/op 975.63 us/op 0.94
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 898.21 us/op 945.25 us/op 0.95
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 2.3856 ms/op 2.3955 ms/op 1.00
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 1.7197 ms/op 1.4876 ms/op 1.16
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 4.0240 ms/op 3.9663 ms/op 1.01
Tree 40 250000 create 348.52 ms/op 335.93 ms/op 1.04
Tree 40 250000 get(125000) 194.43 ns/op 186.77 ns/op 1.04
Tree 40 250000 set(125000) 970.42 ns/op 1.0307 us/op 0.94
Tree 40 250000 toArray() 19.354 ms/op 21.252 ms/op 0.91
Tree 40 250000 iterate all - toArray() + loop 19.844 ms/op 21.202 ms/op 0.94
Tree 40 250000 iterate all - get(i) 72.710 ms/op 71.783 ms/op 1.01
MutableVector 250000 create 11.343 ms/op 9.6246 ms/op 1.18
MutableVector 250000 get(125000) 6.6030 ns/op 6.5830 ns/op 1.00
MutableVector 250000 set(125000) 263.39 ns/op 256.69 ns/op 1.03
MutableVector 250000 toArray() 3.0753 ms/op 2.8348 ms/op 1.08
MutableVector 250000 iterate all - toArray() + loop 3.2707 ms/op 3.0453 ms/op 1.07
MutableVector 250000 iterate all - get(i) 1.5482 ms/op 1.5293 ms/op 1.01
Array 250000 create 2.5847 ms/op 2.5692 ms/op 1.01
Array 250000 clone - spread 1.2198 ms/op 1.1627 ms/op 1.05
Array 250000 get(125000) 0.61500 ns/op 0.57500 ns/op 1.07
Array 250000 set(125000) 0.69000 ns/op 0.66500 ns/op 1.04
Array 250000 iterate all - loop 83.279 us/op 83.960 us/op 0.99
effectiveBalanceIncrements clone Uint8Array 300000 30.689 us/op 32.432 us/op 0.95
effectiveBalanceIncrements clone MutableVector 300000 401.00 ns/op 358.00 ns/op 1.12
effectiveBalanceIncrements rw all Uint8Array 300000 167.48 us/op 170.42 us/op 0.98
effectiveBalanceIncrements rw all MutableVector 300000 85.173 ms/op 81.485 ms/op 1.05
phase0 afterProcessEpoch - 250000 vs - 7PWei 120.69 ms/op 113.13 ms/op 1.07
phase0 beforeProcessEpoch - 250000 vs - 7PWei 34.681 ms/op 42.953 ms/op 0.81
altair processEpoch - mainnet_e81889 288.00 ms/op 299.27 ms/op 0.96
mainnet_e81889 - altair beforeProcessEpoch 62.987 ms/op 48.970 ms/op 1.29
mainnet_e81889 - altair processJustificationAndFinalization 16.242 us/op 17.468 us/op 0.93
mainnet_e81889 - altair processInactivityUpdates 5.3018 ms/op 6.0179 ms/op 0.88
mainnet_e81889 - altair processRewardsAndPenalties 67.608 ms/op 53.221 ms/op 1.27
mainnet_e81889 - altair processRegistryUpdates 2.8680 us/op 3.0820 us/op 0.93
mainnet_e81889 - altair processSlashings 551.00 ns/op 550.00 ns/op 1.00
mainnet_e81889 - altair processEth1DataReset 611.00 ns/op 534.00 ns/op 1.14
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.2060 ms/op 1.3053 ms/op 0.92
mainnet_e81889 - altair processSlashingsReset 4.8580 us/op 4.9780 us/op 0.98
mainnet_e81889 - altair processRandaoMixesReset 4.3880 us/op 4.2230 us/op 1.04
mainnet_e81889 - altair processHistoricalRootsUpdate 668.00 ns/op 641.00 ns/op 1.04
mainnet_e81889 - altair processParticipationFlagUpdates 2.4360 us/op 2.5730 us/op 0.95
mainnet_e81889 - altair processSyncCommitteeUpdates 596.00 ns/op 602.00 ns/op 0.99
mainnet_e81889 - altair afterProcessEpoch 116.86 ms/op 120.21 ms/op 0.97
phase0 processEpoch - mainnet_e58758 318.25 ms/op 315.74 ms/op 1.01
mainnet_e58758 - phase0 beforeProcessEpoch 122.53 ms/op 122.21 ms/op 1.00
mainnet_e58758 - phase0 processJustificationAndFinalization 16.732 us/op 16.893 us/op 0.99
mainnet_e58758 - phase0 processRewardsAndPenalties 54.930 ms/op 54.208 ms/op 1.01
mainnet_e58758 - phase0 processRegistryUpdates 7.6300 us/op 7.7590 us/op 0.98
mainnet_e58758 - phase0 processSlashings 503.00 ns/op 483.00 ns/op 1.04
mainnet_e58758 - phase0 processEth1DataReset 521.00 ns/op 473.00 ns/op 1.10
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 941.52 us/op 993.01 us/op 0.95
mainnet_e58758 - phase0 processSlashingsReset 3.0980 us/op 2.8920 us/op 1.07
mainnet_e58758 - phase0 processRandaoMixesReset 4.3820 us/op 4.3850 us/op 1.00
mainnet_e58758 - phase0 processHistoricalRootsUpdate 608.00 ns/op 630.00 ns/op 0.97
mainnet_e58758 - phase0 processParticipationRecordUpdates 3.7310 us/op 4.2210 us/op 0.88
mainnet_e58758 - phase0 afterProcessEpoch 96.427 ms/op 98.475 ms/op 0.98
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.2126 ms/op 1.2350 ms/op 0.98
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.3661 ms/op 1.4288 ms/op 0.96
altair processInactivityUpdates - 250000 normalcase 19.509 ms/op 21.453 ms/op 0.91
altair processInactivityUpdates - 250000 worstcase 21.768 ms/op 20.043 ms/op 1.09
phase0 processRegistryUpdates - 250000 normalcase 6.0970 us/op 6.9780 us/op 0.87
phase0 processRegistryUpdates - 250000 badcase_full_deposits 227.05 us/op 275.29 us/op 0.82
phase0 processRegistryUpdates - 250000 worstcase 0.5 124.43 ms/op 118.33 ms/op 1.05
altair processRewardsAndPenalties - 250000 normalcase 56.441 ms/op 50.878 ms/op 1.11
altair processRewardsAndPenalties - 250000 worstcase 49.356 ms/op 46.949 ms/op 1.05
phase0 getAttestationDeltas - 250000 normalcase 6.4182 ms/op 6.6371 ms/op 0.97
phase0 getAttestationDeltas - 250000 worstcase 6.5984 ms/op 6.6008 ms/op 1.00
phase0 processSlashings - 250000 worstcase 3.4448 ms/op 3.5035 ms/op 0.98
altair processSyncCommitteeUpdates - 250000 172.77 ms/op 174.36 ms/op 0.99
BeaconState.hashTreeRoot - No change 266.00 ns/op 279.00 ns/op 0.95
BeaconState.hashTreeRoot - 1 full validator 49.561 us/op 52.032 us/op 0.95
BeaconState.hashTreeRoot - 32 full validator 497.56 us/op 525.04 us/op 0.95
BeaconState.hashTreeRoot - 512 full validator 5.6329 ms/op 5.1002 ms/op 1.10
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 59.874 us/op 62.965 us/op 0.95
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 859.32 us/op 924.02 us/op 0.93
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 10.913 ms/op 11.600 ms/op 0.94
BeaconState.hashTreeRoot - 1 balances 48.095 us/op 49.889 us/op 0.96
BeaconState.hashTreeRoot - 32 balances 424.12 us/op 453.94 us/op 0.93
BeaconState.hashTreeRoot - 512 balances 4.2688 ms/op 4.5699 ms/op 0.93
BeaconState.hashTreeRoot - 250000 balances 75.306 ms/op 75.400 ms/op 1.00
aggregationBits - 2048 els - zipIndexesInBitList 16.467 us/op 17.813 us/op 0.92
regular array get 100000 times 32.542 us/op 37.224 us/op 0.87
wrappedArray get 100000 times 32.547 us/op 32.989 us/op 0.99
arrayWithProxy get 100000 times 17.273 ms/op 16.022 ms/op 1.08
ssz.Root.equals 558.00 ns/op 565.00 ns/op 0.99
byteArrayEquals 525.00 ns/op 554.00 ns/op 0.95
shuffle list - 16384 els 6.6520 ms/op 6.8848 ms/op 0.97
shuffle list - 250000 els 97.662 ms/op 101.98 ms/op 0.96
processSlot - 1 slots 8.4100 us/op 9.3360 us/op 0.90
processSlot - 32 slots 1.3010 ms/op 1.4251 ms/op 0.91
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 188.03 us/op 207.39 us/op 0.91
getCommitteeAssignments - req 1 vs - 250000 vc 2.8936 ms/op 2.9250 ms/op 0.99
getCommitteeAssignments - req 100 vs - 250000 vc 4.1278 ms/op 4.2163 ms/op 0.98
getCommitteeAssignments - req 1000 vs - 250000 vc 4.4067 ms/op 4.6123 ms/op 0.96
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 4.7000 ns/op 4.8200 ns/op 0.98
state getBlockRootAtSlot - 250000 vs - 7PWei 905.78 ns/op 836.53 ns/op 1.08
computeProposers - vc 250000 10.276 ms/op 10.694 ms/op 0.96
computeEpochShuffling - vc 250000 101.20 ms/op 103.68 ms/op 0.98
getNextSyncCommittee - vc 250000 174.11 ms/op 178.22 ms/op 0.98

by benchmarkbot/action

@twoeths twoeths enabled auto-merge (squash) March 17, 2023 11:16
@twoeths twoeths merged commit 794ebdd into unstable Mar 18, 2023
@twoeths twoeths deleted the tuyen/fork_choice_spec_slowness branch March 18, 2023 10:52
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.7.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dapplion dapplion dapplion approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@twoeths @wemeetagain @dapplion