BIP: 141 Layer: Consensus (soft fork) Title: Segregated Witness (Consensus layer) Author: Eric Lombrozo <[email protected]> Johnson Lau <[email protected]> Pieter Wuille <[email protected]> Comments-Summary: No comments yet. Comments-URI: https://github.com/bitcoin/bips/wiki/Comments:BIP-0141 Status: Final Type: Standards Track Created: 2015-12-21 License: PD
This BIP defines a new structure called a "witness" that is committed to blocks separately from the transaction merkle tree. This structure contains data required to check transaction validity but not required to determine transaction effects. In particular, scripts and signatures are moved into this new structure.
The witness is committed in a tree that is nested into the block's existing merkle root via the coinbase transaction for the purpose of making this BIP soft fork compatible. A future hard fork can place this tree in its own branch.
The entirety of the transaction's effects are determined by output consumption (spends) and new output creation. Other transaction data, and signatures in particular, are only required to validate the blockchain state, not to determine it.
By removing this data from the transaction structure committed to the transaction merkle tree, several problems are fixed:
- Nonintentional malleability becomes impossible. Since signature data is no longer part of the transaction hash, changes to how the transaction was signed are no longer relevant to transaction identification. As a solution of transaction malleability, this is superior to the canonical signature approach (BIP62):
- It prevents involuntary transaction malleability for any type of scripts, as long as all inputs are signed (with at least one CHECKSIG or CHECKMULTISIG operation)
- In the case of an m-of-n CHECKMULTISIG script, a transaction is malleable only with agreement of m private key holders (as opposed to only 1 private key holder with BIP62)
- It prevents involuntary transaction malleability due to unknown ECDSA signature malleability
- It allows creation of unconfirmed transaction dependency chains without counterparty risk, an important feature for offchain protocols such as the Lightning Network
- Transmission of signature data becomes optional. It is needed only if a peer is trying to validate a transaction instead of just checking its existence. This reduces the size of SPV proofs and potentially improves the privacy of SPV clients as they can download more transactions using the same bandwidth.
- Some constraints could be bypassed with a soft fork by moving part of the transaction data to a structure unknown to current protocol, for example:
- Size of witness could be ignored / discounted when calculating the block size, effectively increasing the block size to some extent
- Hard coded constants, such as maximum data push size (520 bytes) or sigops limit could be reevaluated or removed
- New script system could be introduced without any limitation from the existing script semantic. For example, a new transaction digest algorithm for transaction signature verification is described in BIP143
A new data structure, witness, is defined. Each transaction will have 2 IDs.
Definition of txid remains unchanged: the double SHA256 of the traditional serialization format:
[nVersion][txins][txouts][nLockTime]
A new wtxid is defined: the double SHA256 of the new serialization with witness data:
[nVersion][marker][flag][txins][txouts][witness][nLockTime]
Format of nVersion, txins, txouts, and nLockTime are same as traditional serialization.
The marker MUST be a 1-byte zero value: 0x00.
The flag MUST be a 1-byte non-zero value. Currently, 0x01 MUST be used.
The witness is a serialization of all witness fields of the transaction. Each txin is associated with a witness field. A witness field starts with a var_int to indicate the number of stack items for the txin. It is followed by stack items, with each item starts with a var_int to indicate the length. Witness data is NOT script.
A non-witness program (defined hereinafter) txin MUST be associated with an empty witness field, represented by a 0x00. If all txins are not witness program, a transaction's wtxid is equal to its txid.
A new block rule is added which requires a commitment to the wtxid. The wtxid of coinbase transaction is assumed to be 0x0000....0000.
A witness root hash is calculated with all those wtxid as leaves, in a way similar to the hashMerkleRoot in the block header.
The commitment is recorded in a scriptPubKey of the coinbase transaction. It must be at least 38 bytes, with the first 6-byte of 0x6a24aa21a9ed, that is:
1-byte - OP_RETURN (0x6a) 1-byte - Push the following 36 bytes (0x24) 4-byte - Commitment header (0xaa21a9ed) 32-byte - Commitment hash: Double-SHA256(witness root hash|witness reserved value)
39th byte onwards: Optional data with no consensus meaning
and the coinbase's input's witness must consist of a single 32-byte array for the witness reserved value.
If there are more than one scriptPubKey matching the pattern, the one with highest output index is assumed to be the commitment.
If all transactions in a block do not have witness data, the commitment is optional.
A scriptPubKey (or redeemScript as defined in BIP16/P2SH) that consists of a 1-byte push opcode (one of OP_0,OP_1,OP_2,...,OP_16) followed by a direct data push between 2 and 40 bytes gets a new special meaning. The value of the first push is called the "version byte". The following byte vector pushed is called the "witness program".
In more detail, this means a scriptPubKey or redeemScript which consists of (in order):
- First, byte 0x00 (
OP_0) or any byte between 0x51 (OP_1) and 0x60 (OP_16) inclusive (the version byte). - Then, a byte L between 0x02 (push of 2 bytes) and 0x28 (push of 40 bytes) inclusive.
- Finally, L arbitrary bytes (the witness program).
- Triggered by a
scriptPubKeythat is exactly a push of a version byte, plus a push of a witness program. The scriptSig must be exactly empty or validation fails. ("native witness program") - Triggered when a
scriptPubKeyis a P2SH script, and the BIP16redeemScriptpushed in thescriptSigis exactly a push of a version byte plus a push of a witness program. ThescriptSigmust be exactly a push of the BIP16redeemScriptor validation fails. ("P2SH witness program")
- It is interpreted as a pay-to-witness-public-key-hash (P2WPKH) program.
- The witness must consist of exactly 2 items (≤ 520 bytes each). The first one a signature, and the second one a public key.
- The HASH160 of the public key must match the 20-byte witness program.
- After normal script evaluation, the signature is verified against the public key with CHECKSIG operation. The verification must result in a single TRUE on the stack.
- It is interpreted as a pay-to-witness-script-hash (P2WSH) program.
- The witness must consist of an input stack to feed to the script, followed by a serialized script (
witnessScript). - The
witnessScript(≤ 10,000 bytes) is popped off the initial witness stack. SHA256 of thewitnessScriptmust match the 32-byte witness program. - The
witnessScriptis deserialized, and executed after normal script evaluation with the remaining witness stack (≤ 520 bytes for each stack item). - The script must not fail, and result in exactly a single TRUE on the stack.
If the version byte is 1 to 16, no further interpretation of the witness program or witness stack happens, and there is no size restriction for the witness stack. These versions are reserved for future extensions.[2]
Blocks are currently limited to 1,000,000 bytes (1MB) total size. We change this restriction as follows:
Block weight is defined as Base size * 3 + Total size. (rationale[3])
Base size is the block size in bytes with the original transaction serialization without any witness-related data, as seen by a non-upgraded node.
Total size is the block size in bytes with transactions serialized as described in BIP144, including base data and witness data.
The new rule is block weight ≤ 4,000,000.
Sigops per block is currently limited to 20,000. We change this restriction as follows:
Sigops in the current pubkey script, signature script, and P2SH check script are counted at 4 times their previous value. The sigop limit is likewise quadrupled to ≤ 80,000.
Each P2WPKH input is counted as 1 sigop. In addition, opcodes within a P2WSH witnessScript are counted identically as previously within the P2SH redeemScript. That is, CHECKSIG is counted as only 1 sigop. When preceded by OP_1 to OP_16 CHECKMULTISIG is counted as 1 to 16 sigops respectively, otherwise it is counted as 20 sigops. This rule applies to both native witness program and P2SH witness program.
The following definitions are not used for consensus limits, but are suggested to provide language consistent with the terminology introduced above.
Transaction weight is defined as Base transaction size * 3 + Total transaction size (ie. the same method as calculating Block weight from Base size and Total size).
Virtual transaction size is defined as Transaction weight / 4 (rounded up to the next integer).
Base transaction size is the size of the transaction serialised with the witness data stripped.
Total transaction size is the transaction size in bytes serialized as described in BIP144, including base data and witness data.
Despite that the script language for P2WPKH and P2WSH looks very similar to pre-segregated witness script, there are several notable differences. Users MUST NOT assume that a script spendable in pre-segregated witness system would also be spendable as a P2WPKH or P2WSH script. Before large-scale deployment in the production network, developers should test the scripts on testnet with the default relay policy turned on, and with a small amount of money after BIP141 is activated on mainnet.
A major difference at consensus level is described in BIP143, as a new transaction digest algorithm for signature verification in version 0 witness program.
Three relay and mining policies are also included in the first release of segregated witness at reference implementation version 0.13.1. Softforks based on these policies are likely to be proposed in the near future. To avoid indefinite delay in transaction confirmation and permanent fund loss in a potential softfork, users MUST observe the new semantics carefully:
- Only compressed public keys are accepted in P2WPKH and P2WSH (See BIP143)
- The argument of OP_IF/NOTIF in P2WSH must be minimal[4]
- Signature(s) must be null vector(s) if an OP_CHECKSIG or OP_CHECKMULTISIG is failed (for both pre-segregated witness script and P2WSH. See BIP146)
The following example is a version 0 pay-to-witness-public-key-hash (P2WPKH):
witness: <signature> <pubkey>
scriptSig: (empty)
scriptPubKey: 0 <20-byte-key-hash>
(0x0014{20-byte-key-hash})
The '0' in scriptPubKey indicates the following push is a version 0 witness program. The length of the witness program indicates that it is a P2WPKH type. The witness must consist of exactly 2 items. The HASH160 of the pubkey in witness must match the witness program.
The signature is verified as
<signature> <pubkey> CHECKSIG
Comparing with a traditional P2PKH output, the P2WPKH equivalent occupies 3 less bytes in the scriptPubKey, and moves the signature and public key from scriptSig to witness.
The following example is the same P2WPKH, but nested in a BIP16 P2SH output.
witness: <signature> <pubkey>
scriptSig: <0 <20-byte-key-hash>>
(0x160014{20-byte-key-hash})
scriptPubKey: HASH160 <20-byte-script-hash> EQUAL
(0xA914{20-byte-script-hash}87)
The only item in scriptSig is hashed with HASH160, compared against the 20-byte-script-hash in scriptPubKey, and interpreted as:
0 <20-byte-key-hash>
The public key and signature are then verified as described in the previous example.
Comparing with the previous example, the scriptPubKey is 1 byte bigger and the scriptSig is 23 bytes bigger. Although a nested witness program is less efficient, its payment address is fully transparent and backward compatible for all Bitcoin reference client since version 0.6.0.
The following example is an 1-of-2 multi-signature version 0 pay-to-witness-script-hash (P2WSH).
witness: 0 <signature1> <1 <pubkey1> <pubkey2> 2 CHECKMULTISIG>
scriptSig: (empty)
scriptPubKey: 0 <32-byte-hash>
(0x0020{32-byte-hash})
The '0' in scriptPubKey indicates the following push is a version 0 witness program. The length of the witness program indicates that it is a P2WSH type. The last item in the witness (the "witnessScript") is popped off, hashed with SHA256, compared against the 32-byte-hash in scriptPubKey, and deserialized:
1 <pubkey1> <pubkey2> 2 CHECKMULTISIG
The script is executed with the remaining data from witness:
0 <signature1> 1 <pubkey1> <pubkey2> 2 CHECKMULTISIG
P2WSH allows maximum script size of 10,000 bytes, as the 520-byte push limit is bypassed.
The scriptPubKey occupies 34 bytes, as opposed to 23 bytes of BIP16 P2SH. The increased size improves security against possible collision attacks, as 280 work is not infeasible anymore (By the end of 2015, 284 hashes have been calculated in Bitcoin mining since the creation of Bitcoin). The spending script is same as the one for an equivalent BIP16 P2SH output but is moved to witness.
The following example is the same 1-of-2 multi-signature P2WSH script, but nested in a BIP16 P2SH output.
witness: 0 <signature1> <1 <pubkey1> <pubkey2> 2 CHECKMULTISIG>
scriptSig: <0 <32-byte-hash>>
(0x220020{32-byte-hash})
scriptPubKey: HASH160 <20-byte-hash> EQUAL
(0xA914{20-byte-hash}87)
The only item in scriptSig is hashed with HASH160, compared against the 20-byte-hash in scriptPubKey, and interpreted as:
0 <32-byte-hash>
The P2WSH witnessScript is then executed as described in the previous example.
Comparing with the previous example, the scriptPubKey is 11 bytes smaller (with reduced security) while witness is the same. However, it also requires 35 bytes in scriptSig.
The new commitment in coinbase transaction is a hash of the witness root hash and a witness reserved value. The witness reserved value currently has no consensus meaning, but in the future allows new commitment values for future softforks. For example, if a new consensus-critical commitment is required in the future, the commitment in coinbase becomes:
Double-SHA256(Witness root hash|Hash(new commitment|witness reserved value))
For backward compatibility, the Hash(new commitment|witness reserved value) will go to the coinbase witness, and the witness reserved value will be recorded in another location specified by the future softfork. Any number of new commitment could be added in this way.
Any commitments that are not consensus-critical to Bitcoin, such as merge-mining, MUST NOT use the witness reserved value to preserve the ability to do upgrades of the Bitcoin consensus protocol.
The optional data space following the commitment also leaves room for metadata of future softforks, and MUST NOT be used for other purpose.
Segregated witness fixes the problem of transaction malleability fundamentally, which enables the building of unconfirmed transaction dependency chains in a trust-free manner.
Two parties, Alice and Bob, may agree to send certain amount of Bitcoin to a 2-of-2 multisig output (the "funding transaction"). Without signing the funding transaction, they may create another transaction, time-locked in the future, spending the 2-of-2 multisig output to third account(s) (the "spending transaction"). Alice and Bob will sign the spending transaction and exchange the signatures. After examining the signatures, they will sign and commit the funding transaction to the blockchain. Without further action, the spending transaction will be confirmed after the lock-time and release the funding according to the original contract. It also retains the flexibility of revoking the original contract before the lock-time, by another spending transaction with shorter lock-time, but only with mutual-agreement of both parties.
Such setups are not possible with BIP62 as the malleability fix, since the spending transaction could not be created without both parties first signing the funding transaction. If Alice reveals the funding transaction signature before Bob does, Bob is able to lock up the funding indefinitely without ever signing the spending transaction.
Unconfirmed transaction dependency chain is a fundamental building block of more sophisticated payment networks, such as duplex micropayment channel and the Lightning Network, which have the potential to greatly improve the scalability and efficiency of the Bitcoin system.
Bitcoin right now only has two real security models. A user either runs a full-node which validates every block with all rules in the system, or a SPV (Simple Payment Verification) client which only validates the headers as a proof of publication of some transactions. The Bitcoin whitepaper suggested that SPV nodes may accept alerts from full nodes when they detect an invalid block, prompting the SPV node to download the questioned blocks and transactions for validation. This approach, however, could become a DoS attack vector as there is virtually no cost to generate a false alarm. An alarm must come with a compact, yet deterministic fraud proof.
In the current Bitcoin protocol, it is possible to generate compact fraud proof for almost all rules except a few:
- It is not possible to prove a miner has introduced too many Bitcoins in the coinbase transaction outputs without showing the whole block itself and all input transactions.
- It is not possible to prove the violation of any block specific constraints, such as size and sigop limits, without showing the whole block (and all input transactions in the case of sigop limit)
- It is not possible to prove the spending of a non-existing input without showing all transaction IDs in the blockchain way back to the genesis block.
- Sum trees for transaction fee can be committed making it possible to construct short proofs that the miner does not add excessive fees to the coinbase transaction. Similar for the block size and sigop count limit.
- Backlinks for the outputs spent by the transaction's inputs can be provided. These backlinks consist of a block hash and an offset that thin clients can easily query and check to verify that the outputs exist.
Since a version byte is pushed before a witness program, and programs with unknown versions are always considered as anyone-can-spend script, it is possible to introduce any new script system with a soft fork. The witness as a structure is not restricted by any existing script semantics and constraints, the 520-byte push limit in particular, and therefore allows arbitrarily large scripts and signatures.
Examples of new script systems include Schnorr signatures, which reduce the size of multisig transactions dramatically; Lamport signatures, which are quantum computing resistant; and Merklized abstract syntax trees, which allow very compact witnesses for conditional scripts with extreme complexity.
Currently there is only one nLockTime field in a transaction and all inputs must share the same value. BIP68 enables per-input relative-lock-time using the nSequence field, however, with a limited lock-time period and resolution.
With a soft fork, it is possible to introduce a separate witness structure to allow per-input lock-time and relative-lock-time, and a new script system that could sign and manipulate the new data (like BIP65 and BIP112).
As a soft fork, older software will continue to operate without modification. Non-upgraded nodes, however, will not see nor validate the witness data and will consider all witness programs as anyone-can-spend scripts (except a few edge cases where the witness programs are equal to 0, which the script must fail). Wallets should always be wary of anyone-can-spend scripts and treat them with suspicion. Non-upgraded nodes are strongly encouraged to upgrade in order to take advantage of the new features.
What a non-upgraded wallet can do
- Receiving bitcoin from non-upgraded and upgraded wallets
- Sending bitcoin to non-upgraded and upgraded wallets with traditional P2PKH address (without any benefit of segregated witness)
- Sending bitcoin to upgraded wallets using a P2SH address
- Sending bitcoin to upgraded wallets using a native witness program through BIP70 payment protocol
- Validating segregated witness transaction. It assumes such a transaction is always valid
This BIP will be deployed by "version bits" BIP9 with the name "segwit" and using bit 1.
For Bitcoin mainnet, the BIP9 starttime will be midnight 15 November 2016 UTC (Epoch timestamp 1479168000) and BIP9 timeout will be midnight 15 November 2017 UTC (Epoch timestamp 1510704000).
For Bitcoin testnet, the BIP9 starttime will be midnight 1 May 2016 UTC (Epoch timestamp 1462060800) and BIP9 timeout will be midnight 1 May 2017 UTC (Epoch timestamp 1493596800).
Special thanks to Gregory Maxwell for originating many of the ideas in this BIP and Luke-Jr for figuring out how to deploy this as a soft fork.
- ^ For example, a scriptPubKey with OP_0 followed by a 40-byte non-zero data push will fail due to incorrect program size. However, a scriptPubKey with OP_0 followed by a 41-byte non-zero data push will pass, since it is not considered to be a witness program
- ^ For backward compatibility, for any version byte from 0 to 16, the script must fail if the witness program has a
CastToBoolvalue of zero. However, having a hash like this is a successful preimage attack against the hash function, and the risk is negligible. - ^ Rationale of using a single composite constraint, instead of two separate limits such as 1MB base data and 3MB witness data: Using two separate limits would make mining and fee estimation nearly impossible. Miners would need to solve a complex non-linear optimization problem to find the set of transactions that maximize fees given both constraints, and wallets would not be able to know what to pay as it depends on which of the two conditions is most constrained by the time miners try to produce blocks with their transactions in. Another problem with such an approach is freeloading. Once a set of transactions hit the base data 1MB constraint, up to 3MB extra data could be added to the witness by just minimally increasing the fee. The marginal cost for extra witness space effectively becomes zero in that case.
- ^ https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2016-August/013014.html
- BIP16 Pay to Script Hash
- BIP143 Transaction Signature Verification for Version 0 Witness Program
- BIP144 Segregated Witness (Peer Services)
- BIP173 Base32 address format for native v0-16 witness outputs
This document is placed in the public domain.