Manually test the getblocktemplate RPC creates valid blocks

[oxarbitrage]

Our final goal is to support mining pools to use zebra for their operations. This requires multiple rpc methods to be present.

There is a discussion open to exchange ideas on this.

Details

From #5804 (comment)

I think we can do testing using a block proposal, which is a block without a proof of work.

Here's how we could do that:

• Finish all the required getblocktemplate RPC fields
• Create a block header from the RPC fields, with a zeroed-out equihash solution, and serialize it
• Create the block transactions by combining all the coinbasetxn and transaction data fields
• Send that block to zcashd using `getblocktemplate '{ "mode": "proposal", "data": "(hex data)" }'

zcashd has proposal support, but it's not documented. Here's the spec:
https://en.bitcoin.it/wiki/BIP_0023#Block_Proposal

Related Work

• Manually test that each mining RPC works correctly #5686
• Manually test the getblocktemplate RPC creates valid blocks #5803
• Manually test that Zcash mining pools can use Zebra to mine blocks on testnet #5934
• Manually test that the Zebra and zcashd mempools are similar #5935

[teor2345]

@oxarbitrage please tag your tickets with A-rpc, so we can find them when we're looking for RPC work.

[teor2345]

Please check your box when you have successfully run the zcash-rpc-block-template-to-proposal script from PR #5944, multiple times on mainnet and testnet:

• @arya2
• @oxarbitrage
• @teor2345

You'll need a synced zcashd instance and a synced Zebra instance. You can also test Zebra template to Zebra proposal if you just have Zebra.

We need to merge a few bug fixes to get the script to work all the time:

• fix(rpc): Fix Merkle root transaction order in getblocktemplate RPC method #5953 (fixes failures with multiple transactions)
• fix(rpc): Calculate getblocktemplate RPC testnet min and max times correctly #5925 (fixes failures on testnet using the min or max time test modes)
• Zebra's block template can contain duplicate transactions #5982 (fixes occasional failures due to duplicate transactions in the template)
• Transaction nLockTIme check (CheckFinalTx) against MTP when adding to mempool possible missed #5984 (fixes occasional failures due to invalid lock times, we haven't seen any of these yet)

[teor2345]

I spent some time increasing the coverage of the test script in #5944, and making it easier to use.

I discovered two bugs during my testing:

• Use the same format as zcashd for getblocktemplate proposal mode errors  #5981
• Zebra's block template can contain duplicate transactions #5982

Bug #5982 is a consensus rule bug that will block finishing off testing. (We can still check for other bugs until it is fixed.)

[teor2345]

I am using these shell scripts to run continuous proposal tests until a test fails:

SUCCESSES=0; while zcash-rpc-block-template-to-proposal 28232 8232 28232; do SUCCESSES=$((SUCCESSES+1)); echo "Successes: $SUCCESSES"; sleep 10; done
SUCCESSES=0; while ZCASH_CLI="zcash-cli -testnet" zcash-rpc-block-template-to-proposal 38232 18232 38232; do SUCCESSES=$((SUCCESSES+1)); echo "Successes: $SUCCESSES"; sleep 10; done

My Zebra nodes are on 28232 and 38232, and my zcashd nodes are on the default ports.

[teor2345]

When there is a chain fork change or a block arrives during proposal creation, a inconclusive-not-best-prevblk error is expected.

[mpguerra]

What are the next steps here? When can we be satisfied that we have fully tested this?

[arya2]

What are the next steps here?

We need to fix the known bugs to see if any checks that come after those fail.

When can we be satisfied that we have fully tested this?

It's succeeding consistently for me on the branch that addresses #5982 so that could be the last bug before we can all run this successfully for a few hours and close this issue. We need to fix #5984 too but that bug doesn't seem visible in this test without submitting a transaction that triggers it first.

[mpguerra]

I think we're done here! 🎉