Fix hot state disk leak

[michaelsproul]

Issue Addressed

Fix an issue whereby the v5.2.0-RC hot database grows indefinitely.

Proposed Changes

The main reason for the unbounded growth is the storage of extra hot states.

I accidentally introduced the bug in:

• In-memory tree states #5533

In that PR, we added logic to both:

1. Store advanced states in the database with a "temporary flag".
2. Delete temporary flags for advanced states during block processing.

The problem is, we were deleting the temporary flags for all advanced states, when we should have only been doing this at skipped slots. For example, with 2 consecutive blocks at slots N and N + 1 we would store (forever) the advanced pre-state for N + 1. Our pruning logic would never delete this state because it was not marked temporary, and it was not found by the prune_abandoned_forks logic because it did not lie on any side chains. To fix this bug, we now only delete the temporary flag when a skipped slot occurs.

In addition to this, states with temporary flags were previously only deleted on startup. To allow pruning of these states without restaring, I've ported over some of the state pruning code from full tree-states, which iterates the HotStateSummarys in the database and deletes all states older than the split.slot (or equal to the split slot with a different state root). This is safe even in the case where the split block lies at a slot prior to the split slot, because the database only stores the split slot's advanced (epoch-aligned) state in this case, and we do not risk deleting a canonical state by deleting states with slot < split.slot.

Thanks to @antondlr for noticing the state growth on the testnet deployment.

[michaelsproul]

Seems like some of the tests are unhappy. I will look into it tomorrow.

[michaelsproul]

Compared to the pruning in tree-states, this PR will prune around an epoch "behind", i.e. when updating the split for a new finalization, we delete states that are prior to the current split (old finalization). I think this is acceptable, as it still prevents state growth. It's also simpler to implement, as the pruning method doesn't need to know which state roots are canonical, as in:

lighthouse/beacon_node/beacon_chain/src/migrate.rs

Lines 759 to 768 in b5f2761

	if summary.slot <= new_finalized_slot {
	// If state root doesn't match state root from canonical chain, then delete.
	// We may also find older states here that should have been deleted by `migrate_db`
	// but weren't due to wonky I/O atomicity.
	if newly_finalized_chain
	.get(&summary.slot)
	.map_or(true, |(_, canonical_state_root)| {
	state_root != Hash256::from(*canonical_state_root)
	})
	{

[jimmygchen]

I'm not very familiar with the code here but changes look reasonable and I can't spot any flaws! Looks good to me 👍

[michaelsproul]

@Mergifyio queue

[mergify]

queue

✅ The pull request has been merged automatically

The pull request has been merged automatically at 8762d82