Blob decoupling

[realbigsean]

Description

We have a few streams of work going on in parallel here so I'm hoping to just concretely split up the tasks to make sure features aren't duplicated and so people know who to reach out to

Spec PR: ethereum/consensus-specs#3244

• Gossip decoupling #3991
  • PR: Decouple gossip #3950
  • creating a new "observed cache" to filter duplicates
    • from Pawan:

    • The observed cache has a few more duties than the attestation and sync committee ones
      the cache contains incomplete components of a fully available block (i.e AvailabilityPendingBlock + 4 blobs). Once we get all the components, we essentially go from the AvailabilityPendingBlock to an AvailableBlock which we can process in fork choice

  • create a blob reprocessing flow
  • make the block reprocessing flow dependent on blobs
  • new gossip validation conditions for singular blobs
• Free the blobs in backfill sync #4066
• Free the blobs in range sync #4067
• Blob signing Validator client functionality #3994
  • PR: Blob signing endpoints & validator client blob signing functionality #3979
  • involves creating a cache in the beacon node for full blobs, and only sending blinded blobs to the VC
• Change blob storage structure in database #4099
• Decoupling the by root request #3992
• Blob signing Validator client functionality #3994
  • Add a new blob signing domain (this should be explicit in the new spec)
  • This will eventually involve updates related to the web3signer but I don't think this will be spec'd for a bit
• Availability pending block processing #3995
  • branch: https://github.com/realbigsean/lighthouse/tree/decouple-blobs
• Update crypto interface
  • Issue: Update KZG utils to support decoupled blobs #4076
  • PR: Update kzg interface #4077
• Add KZG commitment EF tests - Implement KZG EF Tests #4274

[jimmygchen]

Devnet v5 spec: https://hackmd.io/@inphi/HJZo4vQGn

Devnet v5:

• Blob decoupling #3989 (this issue)
• Update blob transaction type to 0x03 #4186
• Add corresponding proofs to BlobsBundleV1 ethereum/execution-apis#392 (
  • Engine API change, may not be required if the endpoint gets merged to getPayloadV3 (below)
• Merge getPayloadV3 and getBlobsBundleV1 #4210
  • Engine API change, spec PR not merged, tentative for devnet v5
• Blob signing HTTP endpoints #3993 (optional)
  • Implement get validator block endpoint for EIP-4844 #4075
  • Implement POST beacon block for EIP-4844 #4083

Nice to have:

• Add blob download endpoint (getBlobs) ethereum/beacon-APIs#286
  • Update blobs download endpoint path and BlobSidecar response #4106
  • Add blob indices query parameter

Other:

• Blob signing HTTP endpoints for the builder (blinded) flow #4103

[realbigsean]

I believe @ethDreamer has started on the execution API related changes #4120 and #392

[realbigsean]

Going to close this as we've now decoupled in #4054 and I'd like to consolidate clutter wrt tracking