feat(rpc): add server_definitions method (#4703)

Add a new RPC / WS call for `server_definitions`, which returns an
SDK-compatible `definitions.json` (binary enum definitions) generated by
the server. This enables clients/libraries to dynamically work with new
fields and features, such as ones that may become available on side
chains. Clients query `server_definitions` on a node from the network
they want to work with, and immediately know how to speak that node's
binary "language", even if new features are added to it in the future
(as long as there are no new serialized types that the software doesn't
know how to serialize/deserialize).

Example:

```js
> {"command": "server_definitions"}
< {
    "result": {
        "FIELDS": [
            [
                "Generic",
                {
                    "isSerialized": false,
                    "isSigningField": false,
                    "isVLEncoded": false,
                    "nth": 0,
                    "type": "Unknown"
                }
            ],
            [
                "Invalid",
                {
                    "isSerialized": false,
                    "isSigningField": false,
                    "isVLEncoded": false,
                    "nth": -1,
                    "type": "Unknown"
                }
            ],
            [
                "ObjectEndMarker",
                {
                    "isSerialized": false,
                    "isSigningField": true,
                    "isVLEncoded": false,
                    "nth": 1,
                    "type": "STObject"
                }
            ],
        ...
```

Close #3657

---------

Co-authored-by: Richard Holland <[email protected]>

API-CHANGELOG.md

@@ -26,6 +26,14 @@ In `api_version: 2`, the `signer_lists` field [will be moved](#modifications-to-
 
 The `network_id` field was added in the `server_info` response in version 1.5.0 (2019), but it was not returned in [reporting mode](https://xrpl.org/rippled-server-modes.html#reporting-mode).
 
+## Unreleased
+
+### Additions
+
+Additions are intended to be non-breaking (because they are purely additive).
+
+- `server_definitions`: A new RPC that generates a `definitions.json`-like output that can be used in XRPL libraries.
+
 ## XRP Ledger version 1.12.0
 
 [Version 1.12.0](https://github.com/XRPLF/rippled/releases/tag/1.12.0) was released on Sep 6, 2023.
@@ -46,27 +54,27 @@ Additions are intended to be non-breaking (because they are purely additive).
     - `Account`: The issuer of the asset being clawed back. Must also be the sender of the transaction.
     - `Amount`: The amount being clawed back, with the `Amount.issuer` being the token holder's address.
 - Adds [AMM](https://github.com/XRPLF/XRPL-Standards/discussions/78) ([#4294](https://github.com/XRPLF/rippled/pull/4294), [#4626](https://github.com/XRPLF/rippled/pull/4626)) feature:
-    - Adds `amm_info` API to retrieve AMM information for a given tokens pair.
-    - Adds `AMMCreate` transaction type to create `AMM` instance.
-    - Adds `AMMDeposit` transaction type to deposit funds into `AMM` instance.
-    - Adds `AMMWithdraw` transaction type to withdraw funds from `AMM` instance.
-    - Adds `AMMVote` transaction type to vote for the trading fee of `AMM` instance.
-    - Adds `AMMBid` transaction type to bid for the Auction Slot of `AMM` instance.
-    - Adds `AMMDelete` transaction type to delete `AMM` instance.
-    - Adds `sfAMMID` to `AccountRoot` to indicate that the account is `AMM`'s account. `AMMID` is used to fetch `ltAMM`.
-    - Adds `lsfAMMNode` `TrustLine` flag to indicate that one side of the `TrustLine` is `AMM` account.
-    - Adds `tfLPToken`, `tfSingleAsset`, `tfTwoAsset`, `tfOneAssetLPToken`, `tfLimitLPToken`, `tfTwoAssetIfEmpty`,
-      `tfWithdrawAll`, `tfOneAssetWithdrawAll` which allow a trader to specify different fields combination 
-      for `AMMDeposit` and `AMMWithdraw` transactions.
-    - Adds new transaction result codes:
-       - tecUNFUNDED_AMM: insufficient balance to fund AMM. The account does not have funds for liquidity provision.
-       -  tecAMM_BALANCE: AMM has invalid balance. Calculated balances greater than the current pool balances.
-       -  tecAMM_FAILED: AMM transaction failed. Fails due to a processing failure.
-       -  tecAMM_INVALID_TOKENS: AMM invalid LP tokens. Invalid input values, format, or calculated values.
-       -  tecAMM_EMPTY: AMM is in empty state. Transaction expects AMM in non-empty state (LP tokens > 0).
-       -  tecAMM_NOT_EMPTY: AMM is not in empty state. Transaction expects AMM in empty state (LP tokens == 0).
-       -  tecAMM_ACCOUNT: AMM account. Clawback of AMM account.
-       -  tecINCOMPLETE: Some work was completed, but more submissions required to finish. AMMDelete partially deletes the trustlines.
+  - Adds `amm_info` API to retrieve AMM information for a given tokens pair.
+  - Adds `AMMCreate` transaction type to create `AMM` instance.
+  - Adds `AMMDeposit` transaction type to deposit funds into `AMM` instance.
+  - Adds `AMMWithdraw` transaction type to withdraw funds from `AMM` instance.
+  - Adds `AMMVote` transaction type to vote for the trading fee of `AMM` instance.
+  - Adds `AMMBid` transaction type to bid for the Auction Slot of `AMM` instance.
+  - Adds `AMMDelete` transaction type to delete `AMM` instance.
+  - Adds `sfAMMID` to `AccountRoot` to indicate that the account is `AMM`'s account. `AMMID` is used to fetch `ltAMM`.
+  - Adds `lsfAMMNode` `TrustLine` flag to indicate that one side of the `TrustLine` is `AMM` account.
+  - Adds `tfLPToken`, `tfSingleAsset`, `tfTwoAsset`, `tfOneAssetLPToken`, `tfLimitLPToken`, `tfTwoAssetIfEmpty`,
+    `tfWithdrawAll`, `tfOneAssetWithdrawAll` which allow a trader to specify different fields combination
+    for `AMMDeposit` and `AMMWithdraw` transactions.
+  - Adds new transaction result codes:
+    - tecUNFUNDED_AMM: insufficient balance to fund AMM. The account does not have funds for liquidity provision.
+    - tecAMM_BALANCE: AMM has invalid balance. Calculated balances greater than the current pool balances.
+    - tecAMM_FAILED: AMM transaction failed. Fails due to a processing failure.
+    - tecAMM_INVALID_TOKENS: AMM invalid LP tokens. Invalid input values, format, or calculated values.
+    - tecAMM_EMPTY: AMM is in empty state. Transaction expects AMM in non-empty state (LP tokens > 0).
+    - tecAMM_NOT_EMPTY: AMM is not in empty state. Transaction expects AMM in empty state (LP tokens == 0).
+    - tecAMM_ACCOUNT: AMM account. Clawback of AMM account.
+    - tecINCOMPLETE: Some work was completed, but more submissions required to finish. AMMDelete partially deletes the trustlines.
 
 ## XRP Ledger version 1.11.0
 
@@ -118,6 +126,7 @@ Changes below this point are in development.
 At the time of writing, this version is expected to be introduced in `rippled` version 2.0.
 
 Currently (prior to the release of 2.0), it is available as a "beta" version, meaning it can be enabled with a config setting in `rippled.cfg`:
+
 ```
 [beta_rpc_api]
 1

src/ripple/app/main/Main.cpp

@@ -169,6 +169,7 @@ printHelp(const po::options_description& desc)
            "     peer_reservations_list\n"
            "     ripple ...\n"
            "     ripple_path_find <json> [<ledger>]\n"
+           "     server_definitions [<hash>]\n"
            "     server_info [counters]\n"
            "     server_state [counters]\n"
            "     sign <private_key> <tx_json> [offline]\n"

src/ripple/net/impl/RPCCall.cpp

@@ -1194,6 +1194,20 @@ class RPCParser
         return jvRequest;
     }
 
+    // server_definitions [hash]
+    Json::Value
+    parseServerDefinitions(Json::Value const& jvParams)
+    {
+        Json::Value jvRequest{Json::objectValue};
+
+        if (jvParams.size() == 1)
+        {
+            jvRequest[jss::hash] = jvParams[0u].asString();
+        }
+
+        return jvRequest;
+    }
+
     // server_info [counters]
     Json::Value
     parseServerInfo(Json::Value const& jvParams)
@@ -1255,6 +1269,7 @@ class RPCParser
             {"channel_verify", &RPCParser::parseChannelVerify, 4, 4},
             {"connect", &RPCParser::parseConnect, 1, 2},
             {"consensus_info", &RPCParser::parseAsIs, 0, 0},
+            {"crawl_shards", &RPCParser::parseAsIs, 0, 2},
             {"deposit_authorized", &RPCParser::parseDepositAuthorized, 2, 3},
             {"download_shard", &RPCParser::parseDownloadShard, 2, -1},
             {"feature", &RPCParser::parseFeature, 0, 2},
@@ -1292,14 +1307,14 @@ class RPCParser
              1},
             {"peer_reservations_list", &RPCParser::parseAsIs, 0, 0},
             {"ripple_path_find", &RPCParser::parseRipplePathFind, 1, 2},
+            {"server_definitions", &RPCParser::parseServerDefinitions, 0, 1},
+            {"server_info", &RPCParser::parseServerInfo, 0, 1},
+            {"server_state", &RPCParser::parseServerInfo, 0, 1},
             {"sign", &RPCParser::parseSignSubmit, 2, 3},
             {"sign_for", &RPCParser::parseSignFor, 3, 4},
+            {"stop", &RPCParser::parseAsIs, 0, 0},
             {"submit", &RPCParser::parseSignSubmit, 1, 3},
             {"submit_multisigned", &RPCParser::parseSubmitMultiSigned, 1, 1},
-            {"server_info", &RPCParser::parseServerInfo, 0, 1},
-            {"server_state", &RPCParser::parseServerInfo, 0, 1},
-            {"crawl_shards", &RPCParser::parseAsIs, 0, 2},
-            {"stop", &RPCParser::parseAsIs, 0, 0},
             {"transaction_entry", &RPCParser::parseTransactionEntry, 2, 2},
             {"tx", &RPCParser::parseTx, 1, 4},
             {"tx_account", &RPCParser::parseTxAccount, 1, 7},

src/ripple/protocol/SField.h

@@ -52,43 +52,65 @@ class STInteger;
 class STXChainBridge;
 class STVector256;
 
-enum SerializedTypeID {
-    // special types
-    STI_UNKNOWN = -2,
-    STI_NOTPRESENT = 0,
-
-    // // types (common)
-    STI_UINT16 = 1,
-    STI_UINT32 = 2,
-    STI_UINT64 = 3,
-    STI_UINT128 = 4,
-    STI_UINT256 = 5,
-    STI_AMOUNT = 6,
-    STI_VL = 7,
-    STI_ACCOUNT = 8,
-    // 9-13 are reserved
-    STI_OBJECT = 14,
-    STI_ARRAY = 15,
-
-    // types (uncommon)
-    STI_UINT8 = 16,
-    STI_UINT160 = 17,
-    STI_PATHSET = 18,
-    STI_VECTOR256 = 19,
-    STI_UINT96 = 20,
-    STI_UINT192 = 21,
-    STI_UINT384 = 22,
-    STI_UINT512 = 23,
-    STI_ISSUE = 24,
-    STI_XCHAIN_BRIDGE = 25,
-
-    // high level types
-    // cannot be serialized inside other types
-    STI_TRANSACTION = 10001,
-    STI_LEDGERENTRY = 10002,
-    STI_VALIDATION = 10003,
-    STI_METADATA = 10004,
-};
+#pragma push_macro("XMACRO")
+#undef XMACRO
+
+#define XMACRO(STYPE)                             \
+    /* special types */                           \
+    STYPE(STI_UNKNOWN, -2)                        \
+    STYPE(STI_NOTPRESENT, 0)                      \
+    STYPE(STI_UINT16, 1)                          \
+                                                  \
+    /* types (common) */                          \
+    STYPE(STI_UINT32, 2)                          \
+    STYPE(STI_UINT64, 3)                          \
+    STYPE(STI_UINT128, 4)                         \
+    STYPE(STI_UINT256, 5)                         \
+    STYPE(STI_AMOUNT, 6)                          \
+    STYPE(STI_VL, 7)                              \
+    STYPE(STI_ACCOUNT, 8)                         \
+                                                  \
+    /* 9-13 are reserved */                       \
+    STYPE(STI_OBJECT, 14)                         \
+    STYPE(STI_ARRAY, 15)                          \
+                                                  \
+    /* types (uncommon) */                        \
+    STYPE(STI_UINT8, 16)                          \
+    STYPE(STI_UINT160, 17)                        \
+    STYPE(STI_PATHSET, 18)                        \
+    STYPE(STI_VECTOR256, 19)                      \
+    STYPE(STI_UINT96, 20)                         \
+    STYPE(STI_UINT192, 21)                        \
+    STYPE(STI_UINT384, 22)                        \
+    STYPE(STI_UINT512, 23)                        \
+    STYPE(STI_ISSUE, 24)                          \
+    STYPE(STI_XCHAIN_BRIDGE, 25)                  \
+                                                  \
+    /* high-level types */                        \
+    /* cannot be serialized inside other types */ \
+    STYPE(STI_TRANSACTION, 10001)                 \
+    STYPE(STI_LEDGERENTRY, 10002)                 \
+    STYPE(STI_VALIDATION, 10003)                  \
+    STYPE(STI_METADATA, 10004)
+
+#pragma push_macro("TO_ENUM")
+#undef TO_ENUM
+#pragma push_macro("TO_MAP")
+#undef TO_MAP
+
+#define TO_ENUM(name, value) name = value,
+#define TO_MAP(name, value) {#name, value},
+
+enum SerializedTypeID { XMACRO(TO_ENUM) };
+
+static std::map<std::string, int> const sTypeMap = {XMACRO(TO_MAP)};
+
+#undef XMACRO
+#undef TO_ENUM
+
+#pragma pop_macro("XMACRO")
+#pragma pop_macro("TO_ENUM")
+#pragma pop_macro("TO_MAP")
 
 // constexpr
 inline int
@@ -266,6 +288,12 @@ class SField
     static int
     compare(const SField& f1, const SField& f2);
 
+    static std::map<int, SField const*> const&
+    getKnownCodeToField()
+    {
+        return knownCodeToField;
+    }
+
 private:
     static int num;
     static std::map<int, SField const*> knownCodeToField;

src/ripple/protocol/TER.h

@@ -26,6 +26,7 @@
 #include <optional>
 #include <ostream>
 #include <string>
+#include <unordered_map>
 
 namespace ripple {
 
@@ -644,6 +645,11 @@ isTecClaim(TER x)
     return ((x) >= tecCLAIM);
 }
 
+std::unordered_map<
+    TERUnderlyingType,
+    std::pair<char const* const, char const* const>> const&
+transResults();
+
 bool
 transResultInfo(TER code, std::string& token, std::string& text);
 

src/ripple/protocol/impl/TER.cpp

@@ -20,13 +20,10 @@
 #include <ripple/protocol/TER.h>
 #include <boost/range/adaptor/transformed.hpp>
 #include <type_traits>
-#include <unordered_map>
 
 namespace ripple {
 
-namespace detail {
-
-static std::unordered_map<
+std::unordered_map<
     TERUnderlyingType,
     std::pair<char const* const, char const* const>> const&
 transResults()
@@ -225,12 +222,10 @@ transResults()
     return results;
 }
 
-}  // namespace detail
-
 bool
 transResultInfo(TER code, std::string& token, std::string& text)
 {
-    auto& results = detail::transResults();
+    auto& results = transResults();
 
     auto const r = results.find(TERtoInt(code));
 
@@ -264,7 +259,7 @@ std::optional<TER>
 transCode(std::string const& token)
 {
     static auto const results = [] {
-        auto& byTer = detail::transResults();
+        auto& byTer = transResults();
         auto range = boost::make_iterator_range(byTer.begin(), byTer.end());
         auto tRange = boost::adaptors::transform(range, [](auto const& r) {
             return std::make_pair(r.second.first, r.first);

src/ripple/protocol/jss.h

@@ -321,6 +321,8 @@ JSS(fee_level);             // out: AccountInfo
 JSS(fee_mult_max);          // in: TransactionSign
 JSS(fee_ref);               // out: NetworkOPs, DEPRECATED
 JSS(fetch_pack);            // out: NetworkOPs
+JSS(FIELDS);                // out: RPC server_definitions
+                            // matches definitions.json format
 JSS(first);                 // out: rpc/Version
 JSS(firstSequence);         // out: NodeToShardStatus
 JSS(firstShardIndex);       // out: NodeToShardStatus
@@ -365,6 +367,12 @@ JSS(invalid_API_version);  // out: Many, when a request has an invalid
 JSS(io_latency_ms);        // out: NetworkOPs
 JSS(ip);                   // in: Connect, out: OverlayImpl
 JSS(is_burned);            // out: nft_info (clio)
+JSS(isSerialized);         // out: RPC server_definitions
+                           // matches definitions.json format
+JSS(isSigningField);       // out: RPC server_definitions
+                           // matches definitions.json format
+JSS(isVLEncoded);          // out: RPC server_definitions
+                           // matches definitions.json format
 JSS(issuer);               // in: RipplePathFind, Subscribe,
                            //     Unsubscribe, BookOffers
                            // out: STPathSet, STAmount
@@ -404,6 +412,8 @@ JSS(ledger_index_min);            // in, out: AccountTx*
 JSS(ledger_max);                  // in, out: AccountTx*
 JSS(ledger_min);                  // in, out: AccountTx*
 JSS(ledger_time);                 // out: NetworkOPs
+JSS(LEDGER_ENTRY_TYPES);          // out: RPC server_definitions
+                                  // matches definitions.json format
 JSS(levels);                      // LogLevels
 JSS(limit);                       // in/out: AccountTx*, AccountOffers,
                                   //         AccountLines, AccountObjects
@@ -490,6 +500,7 @@ JSS(node_written_bytes);         // out: GetCounts
 JSS(node_writes_duration_us);    // out: GetCounts
 JSS(node_write_retries);         // out: GetCounts
 JSS(node_writes_delayed);        // out::GetCounts
+JSS(nth);                        // out: RPC server_definitions
 JSS(obligations);                // out: GatewayBalances
 JSS(offer);                      // in: LedgerEntry
 JSS(offers);                     // out: NetworkOPs, AccountOffers, Subscribe
@@ -649,6 +660,12 @@ JSS(transaction);             // in: Tx
 JSS(transaction_hash);        // out: RCLCxPeerPos, LedgerToJson
 JSS(transactions);            // out: LedgerToJson,
                               // in: AccountTx*, Unsubscribe
+JSS(TRANSACTION_RESULTS);     // out: RPC server_definitions
+                              // matches definitions.json format
+JSS(TRANSACTION_TYPES);       // out: RPC server_definitions
+                              // matches definitions.json format
+JSS(TYPES);                   // out: RPC server_definitions
+                              // matches definitions.json format
 JSS(transfer_rate);           // out: nft_info (clio)
 JSS(transitions);             // out: NetworkOPs
 JSS(treenode_cache_size);     // out: GetCounts
@@ -680,7 +697,7 @@ JSS(txr_not_enabled_cnt);     // out: peers with tx reduce-relay disabled count
 JSS(txr_missing_tx_freq);     // out: missing tx frequency average
 JSS(txs);                     // out: TxHistory
 JSS(type);                    // in: AccountObjects
-                              // out: NetworkOPs
+                              // out: NetworkOPs, RPC server_definitions
                               //      OverlayImpl, Logic
 JSS(type_hex);                // out: STPathSet
 JSS(unl);                     // out: UnlList

src/ripple/rpc/handlers/Handlers.h

@@ -127,6 +127,8 @@ doPeerReservationsList(RPC::JsonContext&);
 Json::Value
 doRipplePathFind(RPC::JsonContext&);
 Json::Value
+doServerDefinitions(RPC::JsonContext&);
+Json::Value
 doServerInfo(RPC::JsonContext&);  // for humans
 Json::Value
 doServerState(RPC::JsonContext&);  // for machines