Investigate chain sync bottleneck when reach 1.4M of blocks synced

[EclesioMeloJunior]

Issue summary

• When gossamer reaches the amount of ~1.41M it slows down the sync velocity

[EclesioMeloJunior]

The following pprof output was extracted when Gossamer was in that region

It is possible to notice that the methods handleReadyBlock, handleWorkersResults, processBlockData and processBlockDataWithHeaderAndBody are taking 0s in flat and 19.32s in cum which indicates that a function they call is hanging 99% of the time and we know that the method VerifyBlock is part of the sync flow

If we go deeper in the processBlockDataWithHeaderAndBody we can see that the VerifyBlock is taking 18.82s

To be brief the call chain is:

VerifyBlock -> verifyAuthorshipRight -> verifyBlockEquivocation -> b.blockState.GetBlockHashesBySlot -> GetSlotForBlock -> types.GetSlotFromHeader

and in the end, the function GetSlotFromHeader calls the function DecodeBabePreDigest which uses the scale.Unmarshal to decode the varying data type and as we can notice it is quite expensive.

This process (DecodeBabePreDigest) is made by every descendant of the highest finalized block, and we know that at this point 1.41M in the Westend chain the finalization stopped for some period of time.

Basically, each time we want to import a new block we need to verify it, and since we don't have any finalization happening the number of blocks since the latest finalized block only increases and this creates the bottleneck

[EclesioMeloJunior]

This bottleneck was fixed by #3364