Unify JSON serialization format of transactions

[mDuo13]

There are many API methods that can return transactions. Currently, the return formats are inconsistent. The following table shows where fields are located as of rippled v1.12.0 (unchanged since v1.0.1).

Note: In the following table, the "top" level is with respect to the individual object representing the transaction; for methods that return arrays of transactions, the "top" level is per element of the array. "Transaction Instructions" refer to the canonical fields of the transaction as defined in the transaction format, such as the Account and Flags fields.

Method	Transaction Instructions	Transaction Metadata	hash	ledger_index	inLedger	validated	ledger_hash	Binary blob	date
ledger (with transactions expanded)	Top level	metaData field (JSON) or in meta field¹ (binary)	Top level	Above transaction level	None	Above transaction level	None	tx_blob field¹	None
tx	Top level	meta field	Top level	Top level	Top level	Top level	None	tx field¹	Top level
tx_history (deprecated)	Top level	None	Top level	Top level	Top level	None	None	Not supported	None
account_tx	tx field	meta field	tx field	tx field	tx field	Top level	None	tx_blob field¹	None
transaction_entry (would like to deprecate)	tx_json field	metadata field	tx_json field	Top level	None	Top level	Top level	Not supported	None
sign, sign_for, submit, and submit_multisigned	tx_json field	(N/A)	tx_json field	(N/A)	(N/A)	(N/A)	(N/A)	tx_blob field	None
Streams from subscribe	transaction field	meta field²	Top level	Top level²	None	Top level²	Top level²	Not supported	transaction field
Data API v2 methods	tx field	meta field	Top level	Top level	None	None (Data API only reports validated transaction data)	None	tx¹	Top level (as ISO8601 timestamp, not Ripple time)

¹ Only if the request asked for binary data
² The transactions_proposed stream omits these fields because the transactions' outcomes are not yet final.

Recommendation

Change all of rippled's methods to serialize transactions to JSON in a single consistent format. The format I suggest is essentially the same format the Data API uses, with modifications to accommodate for some formats only the rippled APIs handle. Specifically:

1. Always use tx_blob for binary format. Always use tx_json for JSON format transaction instructions.
2. Only include the canonical transaction instructions ("uppercase fields") and signing fields in the tx_json field with transaction instructions. Move all "lowercase" fields like hash and ledger_index out to the top level of the transaction. (Note: the issuer/currency/value sub-fields of amounts are canonical fields even though they are lowercase.)
3. Always use meta for JSON metadata. Use meta_blob for binary metadata.
4. Remove the inLedger field entirely. (It was a deprecated alias for ledger_index, which is the ledger index of the ledger that includes this transaction.)
5. Add the date field to all transactions that are from a closed ledger. Most of the time this would be the close time of the parent ledger. Make this a UTC ISO8601 timestamp with whole-seconds resolution, for example 2018-07-22T16:37:55Z. Omit this field from transactions that are not in closed ledgers.
6. Add the ledger_hash field to any transactions when pulling them from a closed ledger. Omit this field from transactions that are not in closed ledgers.
7. Add the validated boolean field to all transactions, even when this information is redundant because of context (for example, listing transactions in a validated ledger).

[nbougalis]

I am very supportive of this.

[mDuo13]

This would break backwards compatibility. Perhaps we should put the version method to use.

(More specifically, we should start allowing users to request a specific API version so we can add such breaking changes in a non-disruptive way.)

[mDuo13]

To be crystal clear: the proposed format I am suggesting for all transactions is as follows:

Field	Value	Description
hash	String - Hash	An identifying hash value unique to this transaction, as a hex string.
date	String	(Omitted if from a non-closed ledger) The close time of the ledger that includes this transaction, as a UTC ISO8601 timestamp with whole-seconds resolution, for example 2018-07-22T16:37:55Z.
ledger_index	Number - Ledger Index	(Omitted for the proposed transactions stream) The sequence number of the ledger that includes this transaction.
ledger_hash	String - Hash	(Omitted if from a non-closed ledger) The unique hash of the ledger that includes this transaction.
tx_json	Object	(Omitted if binary requested) The fields of this transaction object aka the transaction instructions, as defined by the Transaction Format.
tx_blob	String - Hex	(Omitted unless binary requested) The binary format of the transaction instructions.
meta	Object	(Omitted if binary requested and omitted from the proposed transactions stream) Metadata about the results of this transaction.
meta_blob	String - Hex	(Omitted unless binary requested; omitted for the proposed transactions stream) The binary format of the transaction metadata.
validated	Boolean	Whether this transaction is in a validated ledger.

(Edit 2023-10-17: I now recommend putting the transaction instructions in tx_json rather than just tx, to reduce confusion with the tx API method, among other things.)

[carlhua]

Consider this for API v2

[intelliot]

Use XLS-22d (basically, just use the api_version field)

[intelliot]

This is a low-priority task for Clio

[ckniffen]

This design is really good. Having just dove into thr typescript definitions for xrpl.js I was very confused.

[zgrguric]

Go for it, there will not be better time to improve consistency than major version bump.

[mDuo13]

Some updates since this is being worked on in XRPLF/rippled#4727 :

• Changed my recommendation from tx to tx_json for the field that contains the transaction instructions and signature in JSON format. This better matches the input format of the signing commands and is a better companion of tx_blob for which is the same thing in binary format.
• After discussing with @intelliot and @Bronek, I think it's worse to change the format of the existing date field than to create a new field for the ISO8601 time string. We agreed on close_time_iso which is a good hint that the timestamp comes from the close time of the ledger where the transaction was applied.
• I think ctid (XLS-37d compact transaction identifier) should be added at the top level, alongside the hash and other fields, in all cases if possible.