refactor: Implement generics for `CheckPoint`, `LocalChain`, and `spk_client` types #1582

LagginTimes · 2024-08-29T09:59:29Z

Implements #1757.

Description

This PR is a step towards header checkpointing for bdk_electrum. The goal is to be able to store whole headers in CheckPoint so they do not have to be re-downloaded. Storing headers this way would be a prerequisite for caching of merkle proofs and for median time passed.

TODO:

Change CheckPoint to take in a generic.
Change LocalChange to take in a generic.
Change spk_client types to take in generics.

Notes to the reviewers

Changelog notice

CheckPoint takes in a generic.
LocalChain and ChangeSet take in generics.
spk_client types can take in generics.

Checklists

All Submissions:

I've signed all my commits
I followed the contribution guidelines
I ran cargo fmt and cargo clippy before committing

crates/core/src/checkpoint.rs

ValuedMammal · 2025-01-05T00:19:58Z

crates/core/src/checkpoint.rs

+struct CPInner<B> {
+    /// Block data.
+    block_id: BlockId,
+    /// Data.
+    data: B,


When B is a blockhash, won't this be storing the same block hash twice? 🤔

+1 on this. Maybe data: Option<B> or just keep the height as part of CPInner?

struct CPInner<B> { height: u32, data: B, prev: Option<Arc<CPinner<B>>>, }

Let's look at this from the API/performance perspective.

I think the "killer feature" of this PR is to be able to put Headers in Checkpoints.

Ideally, we want something like this: CheckPoint<Header>. However, with the proposed changes, it will mean we will be hashing the Header every time we want the block hash from the checkpoint (which will be extremely frequent).

To fix this, we can introduce another data type HeaderWithHash - which, as the name suggests, contains the Header and the BlockHash together. I'm honestly not a fan of this as it becomes clunky.

Since memory is cheap, I think it is okay to duplicate the BlockHash.

If we really don't want to duplicate the BlockHash, another solution would be to use an enum or option internally. It's honestly not pretty and we need to worry about unwraps and how to store the data. Not a fan.

HeaderWithHash sounds reasonable. (or HashedHeader)

crates/core/src/checkpoint.rs

evanlinjin · 2024-09-13T05:26:26Z

crates/chain/tests/test_local_chain.rs

@@ -380,6 +381,88 @@ fn local_chain_insert_block() {
    }
 }

+#[test]
+fn local_chain_insert_header() {


What do you think about constructing an array of consecutive headers first?

Then test cases can be constructed from a subset from this array.

The benefits of this is more correctness. The block hashes line up. The header-to-insert can be constructed from the previous block hash.

evanlinjin · 2025-01-23T11:27:47Z

crates/core/src/checkpoint.rs

+struct CPInner<B> {
+    /// Block data.
+    block_id: BlockId,
+    /// Data.
+    data: B,


Let's look at this from the API/performance perspective.

I think the "killer feature" of this PR is to be able to put Headers in Checkpoints.

Ideally, we want something like this: CheckPoint<Header>. However, with the proposed changes, it will mean we will be hashing the Header every time we want the block hash from the checkpoint (which will be extremely frequent).

To fix this, we can introduce another data type HeaderWithHash - which, as the name suggests, contains the Header and the BlockHash together. I'm honestly not a fan of this as it becomes clunky.

Since memory is cheap, I think it is okay to duplicate the BlockHash.

notmandatory · 2025-01-24T18:42:35Z

Since this one is still having design discussions can I move it out of the 1.1 milestone?

LagginTimes marked this pull request as draft August 29, 2024 09:59

LagginTimes changed the title ~~refactor(core): CheckPoint takes a generic~~ refactor(core): Implement generics for CheckPoint and LocalChain Aug 29, 2024

notmandatory assigned LagginTimes Aug 30, 2024

LagginTimes force-pushed the generic_checkpoint branch 3 times, most recently from a899049 to 7c13781 Compare September 2, 2024 02:53

evanlinjin reviewed Sep 2, 2024

View reviewed changes