TxQ developer docs, and faster ledger size growth

cfg/rippled-example.cfg

@@ -538,6 +538,20 @@
 #       into the ledger at the minimum required fee before the required
 #       fee escalates. Default: no maximum.
 #
+#   normal_consensus_increase_percent = <number>
+#
+#       (Optional) When the ledger has more transactions than "expected",
+#       and performance is humming along nicely, the expected ledger size
+#       is updated to the previous ledger size plus this percentage.
+#       Default: 20
+#
+#   slow_consensus_decrease_percent = <number>
+#
+#       (Optional) When consensus takes longer than appropriate, the
+#       expected ledger size is updated to the minimum of the previous
+#       ledger size or the "expected" ledger size minus this percentage.
+#       Default: 50
+#
 #   maximum_txn_per_account = <number>
 #
 #       Maximum number of transactions that one account can have in the

docs/source.dox

@@ -105,6 +105,9 @@ WARN_LOGFILE           =
 #---------------------------------------------------------------------------
 INPUT                  = \
 \
+    ../src/ripple/app/misc/TxQ.h \
+    ../src/ripple/app/tx/apply.h \
+    ../src/ripple/app/tx/applySteps.h \
     ../src/ripple/protocol/STObject.h \
     ../src/ripple/protocol/JsonFields.h \
     ../src/test/jtx/AbstractClient.h \

src/ripple/app/misc/FeeEscalation.md

@@ -26,17 +26,24 @@ can get into an open ledger for that base fee level. The limit
 will vary based on the [health](#consensus-health) of the
 consensus process, but will be at least [5](#other-constants).
   * If consensus stays [healthy](#consensus-health), the limit will
-  be the max of the current limit or the number of transactions in
-  the validated ledger until it gets to [50](#other-constants), at
-  which point, the limit will be the largest number of transactions
+  be the max of the number of transactions in the validated ledger
+  plus [20%](#other-constants) or the current limit until it gets
+  to [50](#other-constants), at which point, the limit will be the
+  largest number of transactions plus [20%](#other-constants)
   in the last [20](#other-constants) validated ledgers which had
-  more than 50 transactions. Any time the limit decreases (ie. a
-  large ledger is no longer recent), the limit will decrease to the
-  new largest value by 10% each time the ledger has more than 50
-  transactions.
+  more than [50](#other-constants) transactions. Any time the limit
+  decreases (i.e. a large ledger is no longer recent), the limit will
+  decrease to the new largest value by 10% each time the ledger has
+  more than 50 transactions.
   * If consensus does not stay [healthy](#consensus-health),
-  the limit will clamp down to the smaller of [50](#other-constants)
-  or the number of transactions in the validated ledger.
+  the limit will clamp down to the smaller of the number of
+  transactions in the validated ledger minus [50%](#other-constants)
+  or the previous limit minus [50%](#other-constants).
+  * The intended effect of these mechanisms is to allow as many base fee
+  level transactions to get into the ledger as possible while the
+  network is [healthy](#consensus-health), but to respond quickly to
+  any condition that makes it [unhealthy](#consensus-health), including,
+  but not limited to, malicious attacks.
 3. Once there are more transactions in the open ledger than indicated
 by the limit, the required fee level jumps drastically.
   * The formula is `( lastLedgerMedianFeeLevel *
@@ -57,12 +64,12 @@ in the fee escalation formula for the next open ledger.
   * Continuing the example above, if ledger consensus completes with
   only those 20 transactions, and all of those transactions paid the
   minimum required fee at each step, the limit will be adjusted from
-  6 to 20, and the `lastLedgerMedianFeeLevel` will be about 322,000,
+  6 to 24, and the `lastLedgerMedianFeeLevel` will be about 322,000,
   which is 12,600 drops for a
   [reference transaction](#reference-transaction).
-  * This will cause the first 21 transactions only require 10
-  drops, but the 22nd transaction will require
-  a level of about 355,000 or about 13,800 drops.
+  * This will only require 10 drops for the first 25 transactions,
+  but the 26th transaction will require a level of about 349,150
+  or about 13,649 drops.
 
 * This example assumes a cold-start scenario, with a single, possibly
 malicious, user willing to pay arbitrary amounts to get transactions
@@ -99,8 +106,8 @@ external transactions, transactions are applied from the queue to the
 ledger from highest [fee level](#fee-level) to lowest. These transactions
 count against the open ledger limit, so the required [fee level](#fee-level)
 may start rising during this process.
-3. Once the queue is empty, or required the [fee level](#fee-level)
-jumps too high for the remaining transactions in the queue, the ledger
+3. Once the queue is empty, or the required [fee level](#fee-level)
+rises too high for the remaining transactions in the queue, the ledger
 is opened up for normal transaction processing.
 4. A transaction in the queue can stay there indefinitely in principle,
 but in practice, either
@@ -133,7 +140,7 @@ for the queue if it meets these additional criteria:
   * none of the prior queued transactions affect the ability of subsequent
   transactions to claim a fee.
 
-Currently, there is an additional restriction that the queue can not work with
+Currently, there is an additional restriction that the queue cannot work with
 transactions using the `sfPreviousTxnID` or `sfAccountTxnID` fields.
 `sfPreviousTxnID` is deprecated and shouldn't be used anyway. Future
 development will make the queue aware of `sfAccountTxnID` mechanisms.
@@ -195,6 +202,13 @@ unusable. The "target" value of 50 was chosen so the limit never gets large
 enough to invite abuse, but keeps up if the network stays healthy and
 active. These exact values were chosen experimentally, and can easily
 change in the future.
+* *Expected ledger size growth and reduction percentages*. The growth
+value of 20% was chosen to allow the limit to grow quickly as load
+increases, but not so quickly as to allow bad actors to run unrestricted.
+The reduction value of 50% was chosen to cause the limit to drop
+significantly, but not so drastically that the limit cannot quickly
+recover if the problem is temporary. These exact values were chosen
+experimentally, and can easily change in the future.
 * *Minimum `lastLedgerMedianFeeLevel`*. The value of 500 was chosen to
 ensure that the first escalated fee was more significant and noticable
 than what the default would allow. This exact value was chosen
@@ -223,7 +237,7 @@ the earlier ones fails or is otherwise removed from the queue without
 being applied to the open ledger. The value was chosen arbitrarily, and
 can easily change in the future.
 * *Minimum last ledger sequence buffer*. If a transaction has a
-`LastLedgerSequence` value, and can not be processed into the open
+`LastLedgerSequence` value, and cannot be processed into the open
 ledger, that `LastLedgerSequence` must be at least 2 more than the
 sequence number of the open ledger to be considered for the queue. The
 value was chosen to provide a balance between letting the user control
@@ -251,8 +265,8 @@ neccessary fees for other types of transactions. (E.g. multiply all drop
 values by 5 for a multi-signed transaction with 4 signatures.)
 
 The `fee` result is always instantanteous, and relates to the open
-ledger. Thus, it does not include any sequence number or IDs, and may
-not make sense if rippled is not synced to the network.
+ledger. It includes the sequence number of the current open ledger,
+but may not make sense if rippled is not synced to the network.
 
 Result format:
 ```
@@ -262,6 +276,7 @@ Result format:
       "current_queue_size" : "2", // number of transactions waiting in the queue
       "expected_ledger_size" : "15", // one less than the number of transactions that can get into the open ledger for the base fee.
       "max_queue_size" : "300", // number of transactions allowed into the queue
+      "ledger_current_index" : 123456789, // sequence number of the current open ledger
       "levels" : {
          "reference_level" : "256", // level of a reference transaction. Always 256.
          "minimum_level" : "256", // minimum fee level to get into the queue. If >256, indicates the queue is full.

src/ripple/app/misc/NetworkOPs.cpp

@@ -1609,8 +1609,8 @@ NetworkOPsImp::ServerFeeSummary::operator !=(NetworkOPsImp::ServerFeeSummary con
 
     if(em && b.em)
     {
-        return (em->minFeeLevel != b.em->minFeeLevel ||
-                em->expFeeLevel != b.em->expFeeLevel ||
+        return (em->minProcessingFeeLevel != b.em->minProcessingFeeLevel ||
+                em->openLedgerFeeLevel != b.em->openLedgerFeeLevel ||
                 em->referenceFeeLevel != b.em->referenceFeeLevel);
     }
 
@@ -1653,12 +1653,12 @@ void NetworkOPsImp::pubServer ()
         {
             auto const loadFactor =
                 std::max(static_cast<std::uint64_t>(f.loadFactorServer),
-                    mulDiv(f.em->expFeeLevel, f.loadBaseServer,
+                    mulDiv(f.em->openLedgerFeeLevel, f.loadBaseServer,
                         f.em->referenceFeeLevel).second);
 
             jvObj [jss::load_factor]   = clamp(loadFactor);
-            jvObj [jss::load_factor_fee_escalation] = clamp(f.em->expFeeLevel);
-            jvObj [jss::load_factor_fee_queue] = clamp(f.em->minFeeLevel);
+            jvObj [jss::load_factor_fee_escalation] = clamp(f.em->openLedgerFeeLevel);
+            jvObj [jss::load_factor_fee_queue] = clamp(f.em->minProcessingFeeLevel);
             jvObj [jss::load_factor_fee_reference]
                 = clamp(f.em->referenceFeeLevel);
 
@@ -2195,7 +2195,7 @@ Json::Value NetworkOPsImp::getServerInfo (bool human, bool admin, bool counters)
     auto const loadFactorServer = app_.getFeeTrack().getLoadFactor();
     auto const loadBaseServer = app_.getFeeTrack().getLoadBase();
     auto const loadFactorFeeEscalation = escalationMetrics ?
-        escalationMetrics->expFeeLevel : 1;
+        escalationMetrics->openLedgerFeeLevel : 1;
     auto const loadBaseFeeEscalation = escalationMetrics ?
         escalationMetrics->referenceFeeLevel : 1;
 
@@ -2221,7 +2221,7 @@ Json::Value NetworkOPsImp::getServerInfo (bool human, bool admin, bool counters)
                     max32, loadFactorFeeEscalation));
             info[jss::load_factor_fee_queue] =
                 static_cast<std::uint32_t> (std::min(
-                    max32, escalationMetrics->minFeeLevel));
+                    max32, escalationMetrics->minProcessingFeeLevel));
             info[jss::load_factor_fee_reference] =
                 static_cast<std::uint32_t> (std::min(
                     max32, loadBaseFeeEscalation));
@@ -2258,11 +2258,12 @@ Json::Value NetworkOPsImp::getServerInfo (bool human, bool admin, bool counters)
                 info[jss::load_factor_fee_escalation] =
                     static_cast<double> (loadFactorFeeEscalation) /
                         escalationMetrics->referenceFeeLevel;
-            if (escalationMetrics->minFeeLevel !=
+            if (escalationMetrics->minProcessingFeeLevel !=
                     escalationMetrics->referenceFeeLevel)
                 info[jss::load_factor_fee_queue] =
-                    static_cast<double> (escalationMetrics->minFeeLevel) /
-                        escalationMetrics->referenceFeeLevel;
+                    static_cast<double> (
+                        escalationMetrics->minProcessingFeeLevel) /
+                            escalationMetrics->referenceFeeLevel;
         }
     }