Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

merge-queue: embarking main (036a982) and #5257 together #5382

Closed
wants to merge 120 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
120 commits
Select commit Hold shift + click to select a range
0d43b69
Add a new block commit task and channels, that don't do anything yet
teor2345 Sep 12, 2022
63486a1
Add last_block_hash_sent to the state service, to avoid database acce…
teor2345 Sep 13, 2022
c819fb2
Update last_block_hash_sent regardless of commit errors
teor2345 Sep 13, 2022
5c8af25
Rename a field to StateService.max_queued_finalized_height
teor2345 Sep 13, 2022
5ea30f7
Commit finalized blocks to the state in a separate task
teor2345 Sep 13, 2022
1ba6e5b
Check for panics in the block write task
teor2345 Sep 14, 2022
a34032d
Wait for the block commit task in tests, and check for errors
teor2345 Sep 14, 2022
2479be0
Always run a proptest that sleeps once
teor2345 Sep 14, 2022
725adec
Add extra debugging to state shutdowns
teor2345 Sep 14, 2022
9dd5e07
Work around a RocksDB shutdown bug
teor2345 Sep 14, 2022
bbd44b8
Close the finalized block channel when we're finished with it
teor2345 Sep 15, 2022
71f4a59
Only reset state queue once per error
teor2345 Sep 15, 2022
932d9b4
Update some TODOs
teor2345 Sep 15, 2022
d8c63d3
Add a module doc comment
teor2345 Sep 15, 2022
d051464
Drop channels and check for closed channels in the block commit task
teor2345 Sep 15, 2022
1d5455e
Close state channels and tasks on drop
teor2345 Sep 15, 2022
de6ff83
Remove some duplicate fields across StateService and ReadStateService
teor2345 Sep 15, 2022
406fcc2
Try tweaking the shutdown steps
teor2345 Sep 15, 2022
01a4a79
Update and clarify some comments
teor2345 Sep 16, 2022
863bb74
Clarify another comment
teor2345 Sep 16, 2022
6ede492
Don't try to cancel RocksDB background work on drop
teor2345 Sep 16, 2022
1657f28
Fix up some comments
teor2345 Sep 16, 2022
6a50d0e
Remove some duplicate code
teor2345 Sep 16, 2022
439c3dd
Remove redundant workarounds for shutdown issues
teor2345 Sep 16, 2022
e87dd1d
Remode a redundant channel close in the block commit task
teor2345 Sep 20, 2022
04be50b
Remove a mistaken `!force` shutdown condition
teor2345 Sep 20, 2022
cd9bd54
Remove duplicate force-shutdown code and explain it better
teor2345 Sep 20, 2022
622dd5c
Improve RPC error logging
teor2345 Sep 20, 2022
3e95876
Wait for chain tip updates in the RPC tests
teor2345 Sep 20, 2022
82f355a
Wait 2 seconds for chain tip updates before skipping them
teor2345 Sep 20, 2022
199df99
Remove an unnecessary block_in_place()
teor2345 Sep 20, 2022
f597518
Fix some test error messages that were changed by earlier fixes
teor2345 Sep 21, 2022
4710047
Expand some comments, fix typos
teor2345 Sep 21, 2022
0531cd2
Actually drop children of failed blocks
teor2345 Sep 21, 2022
ef35dea
Explain why we drop descendants of failed blocks
teor2345 Sep 21, 2022
61f33f6
Clarify a comment
teor2345 Sep 23, 2022
8029e74
Wait for chain tip updates in a failing test on macOS
teor2345 Sep 23, 2022
5af44a2
Clean duplicate finalized blocks when the non-finalized state activates
teor2345 Sep 23, 2022
b43c433
Send an error when receiving a duplicate finalized block
teor2345 Sep 23, 2022
6eaafc1
Update checkpoint block behaviour, document its consensus rule
teor2345 Sep 23, 2022
9be4996
Wait for chain tip changes in inbound_block_height_lookahead_limit test
teor2345 Sep 23, 2022
b66ef8f
Wait for the genesis block to commit in the fake peer set mempool tests
teor2345 Sep 26, 2022
83a7f2f
Disable unreliable mempool verification check in the send transaction…
teor2345 Sep 27, 2022
8e137ea
Appease rustfmt
teor2345 Sep 27, 2022
04a5808
Use clear_finalized_block_queue() everywhere that blocks are dropped
teor2345 Sep 27, 2022
96d3966
Document how Finalized and NonFinalized clones are different
teor2345 Sep 27, 2022
90b4089
sends non-finalized blocks to the block write task
arya2 Sep 26, 2022
8e10cfc
passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_i…
arya2 Sep 26, 2022
37cf0cc
Update zebra-state/src/service/write.rs
arya2 Sep 27, 2022
a27aa3f
updates comments, renames send_process_queued, other minor cleanup
arya2 Sep 27, 2022
6edd399
Merge branch 'non-finalized-block-commit-channel' of github.com:Zcash…
arya2 Sep 27, 2022
3703a2e
update assert_block_can_be_validated comment
arya2 Sep 27, 2022
c2659b1
removes `mem` field from StateService
arya2 Sep 28, 2022
bc607c5
removes `disk` field from StateService and updates block_iter to use …
arya2 Sep 28, 2022
f2c458d
updates tests that use the disk to use read_service.db instead
arya2 Sep 28, 2022
7631648
moves best_tip to a read fn and returns finalized & non-finalized sta…
arya2 Sep 28, 2022
db31156
changes `contextual_validity` to get the network from the finalized_s…
arya2 Sep 28, 2022
d6327fa
swaps out StateService with FinalizedState and NonFinalizedState in t…
arya2 Sep 28, 2022
0adfe48
adds NotReadyToBeCommitted error and returns it from validate_and_com…
arya2 Sep 28, 2022
c41f8a5
removes NonFinalizedWriteCmd and calls, moves update_latest_channels …
arya2 Sep 28, 2022
93ed20e
makes parent_errors_map an indexmap
arya2 Sep 28, 2022
52a7258
clears non-finalized block queue when the receiver is dropped and whe…
arya2 Sep 28, 2022
caad6cd
sends non-finalized blocks to the block write task
arya2 Sep 26, 2022
b7e1fd4
passes ZebraDb to commit_new_chain, commit_block, and no_duplicates_i…
arya2 Sep 26, 2022
9be511c
updates comments, renames send_process_queued, other minor cleanup
arya2 Sep 27, 2022
fc5f258
Update zebra-state/src/service/write.rs
arya2 Sep 27, 2022
a8c7cd4
update assert_block_can_be_validated comment
arya2 Sep 27, 2022
64f1004
removes `mem` field from StateService
arya2 Sep 28, 2022
82abb0a
removes `disk` field from StateService and updates block_iter to use …
arya2 Sep 28, 2022
c0805e6
updates tests that use the disk to use read_service.db instead
arya2 Sep 28, 2022
5406d5f
moves best_tip to a read fn and returns finalized & non-finalized sta…
arya2 Sep 28, 2022
eddfd26
changes `contextual_validity` to get the network from the finalized_s…
arya2 Sep 28, 2022
9a32357
swaps out StateService with FinalizedState and NonFinalizedState in t…
arya2 Sep 28, 2022
14460cf
adds NotReadyToBeCommitted error and returns it from validate_and_com…
arya2 Sep 28, 2022
76ceb6b
removes NonFinalizedWriteCmd and calls, moves update_latest_channels …
arya2 Sep 28, 2022
a9d4e05
makes parent_errors_map an indexmap
arya2 Sep 28, 2022
3aad1b7
clears non-finalized block queue when the receiver is dropped and whe…
arya2 Sep 28, 2022
dd48089
Merge branch 'non-finalized-block-commit-channel' of github.com:Zcash…
arya2 Sep 28, 2022
5706784
Merge branch 'main' into non-finalized-block-commit-channel
arya2 Sep 28, 2022
7ac64be
removes duplicate field definitions on StateService that were a resul…
arya2 Sep 28, 2022
6c4c7e8
update NotReadyToBeCommitted error message
arya2 Sep 28, 2022
cbd8cdb
Appear rustfmt
arya2 Sep 28, 2022
580e1bd
Fix doc links
teor2345 Sep 29, 2022
ff53e74
Rename a function to initial_contextual_validity()
teor2345 Sep 29, 2022
0fc7dfe
Do error tasks on Err, and success tasks on Ok
teor2345 Sep 29, 2022
d767001
Simplify parent_error_map truncation
teor2345 Sep 29, 2022
48dbd3c
Rewrite best_tip() to use tip()
teor2345 Sep 29, 2022
5dba5b2
Rename latest_mem() to latest_non_finalized_state()
teor2345 Sep 29, 2022
d868318
Simplify latest_non_finalized_state() using a new WatchReceiver API
teor2345 Sep 29, 2022
2765681
Expand some error messages
teor2345 Sep 29, 2022
22a78cd
Send the result after updating the channels, and document why
teor2345 Sep 29, 2022
17ee713
wait for chain_tip_update before cancelling download in mempool_cance…
arya2 Sep 29, 2022
bf125ac
adds `sent_non_finalized_block_hashes` field to StateService
arya2 Sep 30, 2022
70beeb0
adds batched sent_hash insertions and checks sent hashes in queue_and…
arya2 Oct 3, 2022
b8cc17a
check that the `curr_buf` in SentHashes is not empty before pushing i…
arya2 Oct 3, 2022
a3da5c2
Apply suggestions from code review
arya2 Oct 3, 2022
8ef23e4
Fix rustfmt
teor2345 Oct 3, 2022
6bc77f5
Check for finalized block heights using zs_contains()
teor2345 Oct 3, 2022
9e0076f
Merge branch 'main' into non-finalized-block-commit-channel
mergify[bot] Oct 4, 2022
e447032
Merge branch 'main' into non-finalized-block-commit-channel
mergify[bot] Oct 4, 2022
daa9a0a
adds known_utxos field to SentHashes
arya2 Oct 4, 2022
6e1fb1d
updates comment on SentHashes.add method
arya2 Oct 4, 2022
a999432
Apply suggestions from code review
arya2 Oct 4, 2022
c9ae0b7
return early when there's a duplicate hash in QueuedBlocks.queue inst…
arya2 Oct 5, 2022
8444162
Merge branch 'main' into non-finalized-block-commit-channel
arya2 Oct 5, 2022
0b675cc
Merge branch 'main' into non-finalized-block-commit-channel
mergify[bot] Oct 6, 2022
28fe080
Merge branch 'main' into non-finalized-block-commit-channel
teor2345 Oct 7, 2022
29a8ada
Make finalized UTXOs near the final checkpoint available for full blo…
teor2345 Oct 7, 2022
56cfa72
Replace a checkpoint height literal with the actual config
teor2345 Oct 7, 2022
9c3cd2f
Update mainnet and testnet checkpoints - 7 October 2022
teor2345 Oct 7, 2022
29e765f
Fix some state service init arguments
teor2345 Oct 7, 2022
2e3eaef
Allow more lookahead in the downloader, but less lookahead in the syncer
teor2345 Oct 7, 2022
987d21d
Add the latest config to the tests, and fix the latest config check
teor2345 Oct 7, 2022
a1fcf41
Increase the number of finalized blocks checked for non-finalized blo…
teor2345 Oct 7, 2022
e3290df
fix(log): reduce verbose logs for block commits (#5348)
teor2345 Oct 7, 2022
8eea77a
Increase block verify timeout
teor2345 Oct 7, 2022
7a926b1
Merge branch 'main' into non-finalized-block-commit-channel
teor2345 Oct 11, 2022
74b29be
Work around a known block timeout bug by using a shorter timeout
teor2345 Oct 11, 2022
a0a65b6
Merge branch 'checkpoints-2022-10-07' into non-finalized-block-commit…
teor2345 Oct 11, 2022
cdd40b6
Merge of #5257
mergify[bot] Oct 11, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

27 changes: 18 additions & 9 deletions zebra-consensus/src/chain.rs
Original file line number Diff line number Diff line change
Expand Up @@ -235,15 +235,7 @@ where
let transaction = Buffer::new(BoxService::new(transaction), VERIFIER_BUFFER_BOUND);

// block verification

let list = CheckpointList::new(network);

let max_checkpoint_height = if config.checkpoint_sync {
list.max_height()
} else {
list.min_height_in_range(network.mandatory_checkpoint_height()..)
.expect("hardcoded checkpoint list extends past canopy activation")
};
let (list, max_checkpoint_height) = init_checkpoint_list(config, network);

let tip = match state_service
.ready()
Expand Down Expand Up @@ -275,3 +267,20 @@ where
max_checkpoint_height,
)
}

/// Parses the checkpoint list for `network` and `config`.
/// Returns the checkpoint list and maximum checkpoint height.
pub fn init_checkpoint_list(config: Config, network: Network) -> (CheckpointList, Height) {
// TODO: Zebra parses the checkpoint list twice at startup.
// Instead, cache the checkpoint list for each `network`.
let list = CheckpointList::new(network);

let max_checkpoint_height = if config.checkpoint_sync {
list.max_height()
} else {
list.min_height_in_range(network.mandatory_checkpoint_height()..)
.expect("hardcoded checkpoint list extends past canopy activation")
};

(list, max_checkpoint_height)
}
1 change: 1 addition & 0 deletions zebra-state/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ dirs = "4.0.0"
displaydoc = "0.2.3"
futures = "0.3.24"
hex = "0.4.3"
indexmap = "1.9.1"
itertools = "0.10.5"
lazy_static = "1.4.0"
metrics = "0.20.1"
Expand Down
4 changes: 4 additions & 0 deletions zebra-state/src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,10 @@ pub struct CommitBlockError(#[from] ValidateContextError);
#[non_exhaustive]
#[allow(missing_docs)]
pub enum ValidateContextError {
#[error("block parent not found in any chain")]
#[non_exhaustive]
NotReadyToBeCommitted,

#[error("block height {candidate_height:?} is lower than the current finalized height {finalized_tip_height:?}")]
#[non_exhaustive]
OrphanedBlock {
Expand Down
Loading