Skip to content

Commit

Permalink
chaindag: don't keep backfill block table in memory (#3429)
Browse files Browse the repository at this point in the history 
This PR names and documents the concept of the archive: a range of slots
for which we have degraded functionality in terms of historical access -
in particular:

* we don't support rewinding to states in this range
* we don't keep an in-memory representation of the block dag

The archive de-facto exists in a trusted-node-synced node, but this PR
gives it a name and drops the in-memory digest index.

In order to satisfy `GetBlocksByRange` requests, we ensure that we have
blocks for the entire archive period via backfill. Future versions may
relax this further, adding a "pre-archive" period that is fully pruned.

During by-slot searches in the archive (both for libp2p and rest
requests), an extra database lookup is used to covert the given `slot`
to a `root` - future versions will avoid this using era files which
natively are indexed by `slot`. That said, the lookup is quite
fast compared to the actual block loading given how trivial the table
is - it's hard to measure, even.

A collateral benefit of this PR is that checkpoint-synced nodes will see
100-200MB memory usage savings, thanks to the dropped in-memory cache -
future pruning work will bring this benefit to full nodes as well.

* document chaindag storage architecture and assumptions
* look up parent using block id instead of full block in clearance
(future-proofing the code against a future in which blocks come from era
files)
* simplify finalized block init, always writing the backfill portion to
db at startup (to ensure lookups work as expected)
* preallocate some extra memory for finalized blocks, to avoid immediate
realloc
  • Loading branch information
@arnetheduck
arnetheduck authored Feb 26, 2022
1 parent 92e7e28 commit 40a4c01
Show file tree
Hide file tree
Showing 6 changed files with 215 additions and 138 deletions.
20 changes: 19 additions & 1 deletion beacon_chain/beacon_chain_db.nim
View file
Original file line number Diff line number Diff line change
Expand Up @@ -119,10 +119,28 @@ type
## are stored in a mod-increment pattern across fixed-sized arrays, which
## addresses most of the rest of the BeaconState sizes.

summaries: KvStoreRef # BlockRoot -> BeaconBlockSummary
summaries: KvStoreRef
## BlockRoot -> BeaconBlockSummary - permits looking up basic block
## information via block root - contains only summaries that were valid
## at some point in history - it is however possible that entries exist
## that are no longer part of the finalized chain history, thus the
## cache should not be used to answer fork choice questions - see
## `getHeadBlock` and `finalizedBlocks` instead.
##
## May contain entries for blocks that are not stored in the database.
##
## See `finalizedBlocks` for an index in the other direction.

finalizedBlocks*: FinalizedBlocks
## Blocks that are known to be finalized, per the latest head (v1.7.0+)
## Only blocks that have passed verification, either via state transition
## or backfilling are indexed here - thus, similar to `head`, it is part
## of the inner security ring and is used to answer security questions
## in the chaindag.
##
## May contain entries for blocks that are not stored in the database.
##
## See `summaries` for an index in the other direction.

DbKeyKind = enum
kHashToState
Expand Down
75 changes: 39 additions & 36 deletions beacon_chain/consensus_object_pools/block_clearance.nim
View file
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,7 @@ proc addHeadBlock*(
logScope:
blockRoot = shortLog(signedBlock.root)
blck = shortLog(signedBlock.message)
signature = shortLog(signedBlock.signature)

template blck(): untyped = signedBlock.message # shortcuts without copy
template blockRoot(): untyped = signedBlock.root
Expand All @@ -162,11 +163,10 @@ proc addHeadBlock*(
# happens in the meantime - the block we requested will then be stale
# by the time it gets here.
if blck.slot <= dag.finalizedHead.slot:
let previous = dag.getBlockIdAtSlot(blck.slot)
if previous.isProposed() and blockRoot == previous.bid.root:
# We should not call the block added callback for blocks that already
# existed in the pool, as that may confuse consumers such as the fork
# choice.
let existing = dag.getBlockIdAtSlot(blck.slot)
# The exact slot match ensures we reject blocks that were orphaned in
# the finalized chain
if existing.bid.slot == blck.slot and blockRoot == existing.bid.root:
debug "Duplicate block"
return err(BlockError.Duplicate)

Expand All @@ -188,18 +188,20 @@ proc addHeadBlock*(
# is on is no longer a viable fork candidate - we can't tell which is which
# at this stage, but we can check if we've seen the parent block previously
# and thus prevent requests for it to be downloaded again.
if dag.db.containsBlock(blck.parent_root):
debug "Block unviable due to pre-finalized-checkpoint parent"
let parentId = dag.getBlockId(blck.parent_root)
if parentId.isSome():
debug "Block unviable due to pre-finalized-checkpoint parent",
parentId = parentId.get()
return err(BlockError.UnviableFork)

debug "Block parent unknown or finalized already"
debug "Block parent unknown or finalized already", parentId
return err(BlockError.MissingParent)

if parent.slot >= signedBlock.message.slot:
if parent.slot >= blck.slot:
# A block whose parent is newer than the block itself is clearly invalid -
# discard it immediately
debug "Block with invalid parent",
parentBlock = shortLog(parent)
debug "Block older than parent",
parent = shortLog(parent)

return err(BlockError.Invalid)

Expand Down Expand Up @@ -237,8 +239,8 @@ proc addHeadBlock*(
# A PublicKey or Signature isn't on the BLS12-381 curve
info "Unable to load signature sets",
err = e.error()

return err(BlockError.Invalid)

if not verifier.batchVerify(sigs):
info "Block signature verification failed",
signature = shortLog(signedBlock.signature)
Expand Down Expand Up @@ -274,6 +276,7 @@ proc addBackfillBlock*(
logScope:
blockRoot = shortLog(signedBlock.root)
blck = shortLog(signedBlock.message)
signature = shortLog(signedBlock.signature)
backfill = (dag.backfill.slot, shortLog(dag.backfill.parent_root))

template blck(): untyped = signedBlock.message # shortcuts without copy
Expand All @@ -282,19 +285,18 @@ proc addBackfillBlock*(
let startTick = Moment.now()

if blck.slot >= dag.backfill.slot:
let previous = dag.getBlockIdAtSlot(blck.slot)
if previous.isProposed() and blockRoot == previous.bid.root:
let existing = dag.getBlockIdAtSlot(blck.slot)
if existing.bid.slot == blck.slot and blockRoot == existing.bid.root:
# We should not call the block added callback for blocks that already
# existed in the pool, as that may confuse consumers such as the fork
# choice. While the validation result won't be accessed, it's IGNORE,
# according to the spec.
# choice.
debug "Duplicate block"
return err(BlockError.Duplicate)

# Block is older than finalized, but different from the block in our
# canonical history: it must be from an unviable branch
debug "Block from unviable fork",
finalizedHead = shortLog(dag.finalizedHead),
backfill = shortLog(dag.backfill)
finalizedHead = shortLog(dag.finalizedHead)

return err(BlockError.UnviableFork)

Expand All @@ -306,16 +308,18 @@ proc addBackfillBlock*(
# can happen is when an invalid `--network` parameter is given during
# startup (though in theory, we check that - maybe the database was
# swapped or something?).
fatal "Checkpoint given during initial startup inconsistent with genesis - wrong network used when starting the node?"
fatal "Checkpoint given during initial startup inconsistent with genesis block - wrong network used when starting the node?",
genesis = shortLog(dag.genesis), tail = shortLog(dag.tail),
head = shortLog(dag.head)
quit 1

dag.backfillBlocks[blck.slot.int] = blockRoot
dag.backfill = blck.toBeaconBlockSummary()
dag.db.finalizedBlocks.insert(blck.slot, blockRoot)

notice "Received matching genesis block during backfill, backfill complete"
notice "Received final block during backfill, backfill complete"

# Backfill done - dag.backfill.slot now points to genesis block just like
# it would if we loaded a fully backfilled database - returning duplicate
# it would if we loaded a fully synced database - returning duplicate
# here is appropriate, though one could also call it ... ok?
return err(BlockError.Duplicate)

Expand All @@ -325,14 +329,18 @@ proc addBackfillBlock*(

# If the hash is correct, the block itself must be correct, but the root does
# not cover the signature, which we check next

let proposerKey = dag.validatorKey(blck.proposer_index)
if proposerKey.isNone():
# This cannot happen, in theory, unless the checkpoint state is broken or
# there is a bug in our validator key caching scheme - in order not to
# send invalid attestations, we'll shut down defensively here - this might
# need revisiting in the future.
fatal "Invalid proposer in backfill block - checkpoint state corrupt?"
# We've verified that the block root matches our expectations by following
# the chain of parents all the way from checkpoint. If all those blocks
# were valid, the proposer_index in this block must also be valid, and we
# should have a key for it but we don't: this is either a bug on our from
# which we cannot recover, or an invalid checkpoint state was given in which
# case we're in trouble.
fatal "Invalid proposer in backfill block - checkpoint state corrupt?",
head = shortLog(dag.head), tail = shortLog(dag.tail),
genesis = shortLog(dag.genesis)

quit 1

if not verify_block_signature(
Expand All @@ -342,20 +350,15 @@ proc addBackfillBlock*(
signedBlock.root,
proposerKey.get(),
signedBlock.signature):
info "Block signature verification failed",
signature = shortLog(signedBlock.signature)
info "Block signature verification failed"
return err(BlockError.Invalid)

let sigVerifyTick = Moment.now

dag.putBlock(signedBlock.asTrusted())
dag.db.finalizedBlocks.insert(blck.slot, blockRoot)

# Invariants maintained on startup
doAssert dag.backfillBlocks.lenu64 == dag.tail.slot.uint64
doAssert dag.backfillBlocks.lenu64 > blck.slot.uint64

dag.backfillBlocks[blck.slot.int] = blockRoot
dag.backfill = blck.toBeaconBlockSummary()
dag.db.finalizedBlocks.insert(blck.slot, blockRoot)

let putBlockTick = Moment.now
debug "Block backfilled",
Expand Down
6 changes: 2 additions & 4 deletions beacon_chain/consensus_object_pools/block_dag.nim
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,9 @@

import
chronicles,
../spec/forks

../spec/datatypes/[phase0, altair],
../spec/[helpers]

export chronicles, phase0, altair, helpers
export chronicles, forks

type
BlockId* = object
Expand Down
110 changes: 70 additions & 40 deletions beacon_chain/consensus_object_pools/block_pools_types.nim
View file
Original file line number Diff line number Diff line change
Expand Up @@ -61,80 +61,110 @@ type
data*: BlockRef

ChainDAGRef* = ref object
## Pool of blocks responsible for keeping a DAG of resolved blocks.
## ChainDAG validates, stores and serves chain history of valid blocks
## according to the beacon chain state transtion. From genesis to the
## finalization point, block history is linear - from there, it branches out
## into a dag with several heads, one of which is considered canonical.
##
## It is responsible for the following
## As new blocks are added, new segments of the chain may finalize,
## discarding now unviable candidate histories.
##
## - Handle requests and updates to the "ColdDB" which
## holds the canonical chain.
## - Maintain a direct acyclic graph (DAG) of
## candidate chains from the last
## finalized block.
## In addition to storing blocks, the chaindag also is responsible for
## storing intermediate states in the database that are used to recreate
## chain history at any point in time through a rewinding process that loads
## a snapshots and applies blocks until the desired point in history is
## reached.
##
## When a chain becomes finalized, it is saved in the ColdDB,
## the rejected candidates are discarded and this pool
## is pruned, only keeping the last finalized block.
## Several indices are kept in memory to enable fast lookups - their shape
## and contents somewhat depend on how the chain was instantiated: sync
## from genesis or checkpoint, and therefore, what features we can offer in
## terms of historical replay.
##
## The last finalized block is called the tail block.

# -----------------------------------
# ColdDB - Canonical chain
## Beacuse the state transition is forwards-only, checkpoint sync generally
## allows replaying states from that point onwards - anything earlier
## would require a backfill of blocks and a subsequent replay from genesis.
##
## Era files contain state snapshots along the way, providing arbitrary
## starting points for replay and can be used to frontfill the archive -
## however, they are not used until the contents have been verified via
## parent_root ancestry.
##
## The chain and the various pointers and indices we keep can be seen in
## the following graph: depending on how the chain was instantiated, some
## pointers may overlap and some indices might be empty as a result.
##
## / heads
## /-------* |
## *--------*---------*---------------*--------------*
## | | | | |
## genesis backfill tail finalizedHead head
## | | |
## archive finalizedBlocks forkBlocks
##
## The archive is the the part of finalized history for which we no longer
## recreate states quickly because we don't have a reasonable state to
## start replay from - when starting from a checkpoint, this is the typical
## case - recreating history requires either replaying from genesis or
## providing an earlier checkpoint state.
##
## We do not keep an in-memory index for the archive - instead, lookups are
## made via `BeaconChainDB.finalizedBlocks` which covers the full range from
## `backfill` to `finalizedHead`.

db*: BeaconChainDB
## ColdDB - Stores the canonical chain
## Database of recent chain history as well as the state and metadata
## needed to pick up where we left off after a restart - in particular,
## the DAG and the canonical head are stored here, as well as several
## caches.

validatorMonitor*: ref ValidatorMonitor

# -----------------------------------
# ChainDAGRef - DAG of candidate chains

forkBlocks*: HashSet[KeyedBlockRef]
## root -> BlockRef mapping of blocks still relevant to fork choice, ie
## root -> BlockRef mapping of blocks relevant to fork choice, ie
## those that have not yet been finalized - covers the slots
## `finalizedHead.slot..head.slot` (inclusive)
## `finalizedHead.slot..head.slot` (inclusive) - dag.heads keeps track
## of each potential head block in this table.

finalizedBlocks*: seq[BlockRef]
## Slot -> BlockRef mapping for the canonical chain - use getBlockAtSlot
## to access, generally - covers the slots
## `tail.slot..finalizedHead.slot` (including the finalized head block) -
## indices are thus offset by tail.slot

backfillBlocks*: seq[Eth2Digest]
## Slot -> Eth2Digest, covers genesis.slot..tail.slot - 1 (inclusive)
## Slot -> BlockRef mapping for the finalized portion of the canonical
## chain - use getBlockAtSlot to access
## Covers the slots `tail.slot..finalizedHead.slot` (including the
## finalized head block). Indices offset by `tail.slot`.

genesis*: BlockRef
## The genesis block of the network
## The genesis block of the network

tail*: BlockRef
## The earliest finalized block for which we have a corresponding state -
## when making a replay of chain history, this is as far back as we can
## go - the tail block is unique in that its parent is set to `nil`, even
## in the case where an earlier genesis block exists.
## The earliest finalized block for which we have a corresponding state -
## when making a replay of chain history, this is as far back as we can
## go - the tail block is unique in that its parent is set to `nil`, even
## in the case where an earlier genesis block exists.

backfill*: BeaconBlockSummary
## The backfill points to the oldest block that we have in the database -
## when backfilling, the first block to download is the parent of this block
## The backfill points to the oldest block with an unbroken ancestry from
## dag.tail - when backfilling, we'll move backwards in time starting
## with the parent of this block until we reach `genesis`.

heads*: seq[BlockRef]
## Candidate heads of candidate chains

finalizedHead*: BlockSlot
## The latest block that was finalized according to the block in head
## Ancestors of this block are guaranteed to have 1 child only.
## The latest block that was finalized according to the block in head
## Ancestors of this block are guaranteed to have 1 child only.

# -----------------------------------
# Pruning metadata

lastPrunePoint*: BlockSlot
## The last prune point
## We can prune up to finalizedHead
## The last prune point
## We can prune up to finalizedHead

# -----------------------------------
# Rewinder - Mutable state processing

headState*: StateData
## State given by the head block - must only be updated in `updateHead` -
## always matches dag.head
## State given by the head block - must only be updated in `updateHead` -
## always matches dag.head

epochRefState*: StateData
## State used to produce epochRef instances - must only be used in
Expand Down
Loading

0 comments on commit 40a4c01

Please sign in to comment.