[chain] Charge Write Fee on Allocate + Remove Cold/Warm (#608)

* outline some changes for cold fee charging

* add failing test

* add warm modifications test

* add more TODOs

* add more TODOs

* use storageWrite and storageAllocate

* remove cold/warm from dependencies

* update tstate

* tstate passes tests

* test key suffix logic

* cleaning up chunk handling

* explore rollback -> ops likely broken

* working on cleanup

* passing all tests

* remove unnecessary functions

* add complex test

* add failing tests

* progress

* fixing tests

* tests passing

* don't emit an op for a useless insert

* work on renaming in chain/transaction

* resolve errors in chain/transaction

* chain package compiles

* everything compiles

* update comment for change

* use allocate, not allocations

* integration passing

* tokenvm integration works

* remove junk file

* update frontend + cli

* fix readme

* reach 100% coverage on tstate

README.md

@@ -236,7 +236,7 @@ verify a batch.
 ### Multidimensional Fee Pricing
 Instead of mapping transaction resource usage to a one-dimensional unit (i.e. "gas"
 or "fuel"), the `hypersdk` utilizes five independently parameterized unit dimensions
-(bandwidth, compute, storage[read], storage[create], storage[modify]) to meter
+(bandwidth, compute, storage[read], storage[allocate], storage[write]) to meter
 activity on each `hypervm`. Each unit dimension has a unique metering schedule
 (i.e. how many units each resource interaction costs), target, and max utilization
 per rolling 10 second window.
@@ -280,16 +280,12 @@ GetBaseWarpComputeUnits() uint64
 GetWarpComputeUnitsPerSigner() uint64
 GetOutgoingWarpComputeUnits() uint64
 
-GetColdStorageKeyReadUnits() uint64
-GetColdStorageValueReadUnits() uint64 // per chunk
-GetWarmStorageKeyReadUnits() uint64
-GetWarmStorageValueReadUnits() uint64 // per chunk
-GetStorageKeyCreateUnits() uint64
-GetStorageValueCreateUnits() uint64 // per chunk
-GetColdStorageKeyModificationUnits() uint64
-GetColdStorageValueModificationUnits() uint64 // per chunk
-GetWarmStorageKeyModificationUnits() uint64
-GetWarmStorageValueModificationUnits() uint64 // per chunk
+GetStorageKeyReadUnits() uint64
+GetStorageValueReadUnits() uint64 // per chunk
+GetStorageKeyAllocateUnits() uint64
+GetStorageValueAllocateUnits() uint64 // per chunk
+GetStorageKeyWriteUnits() uint64
+GetStorageValueWriteUnits() uint64 // per chunk
 ```
 
 An example configuration may look something like:
@@ -304,16 +300,12 @@ BaseWarpComputeUnits:      1_024,
 WarpComputeUnitsPerSigner: 128,
 OutgoingWarpComputeUnits:  1_024,
 
-ColdStorageKeyReadUnits:           5,
-ColdStorageValueReadUnits:         2,
-WarmStorageKeyReadUnits:           1,
-WarmStorageValueReadUnits:         1,
-StorageKeyCreateUnits:             20,
-StorageValueCreateUnits:           5,
-ColdStorageKeyModificationUnits:   10,
-ColdStorageValueModificationUnits: 3,
-WarmStorageKeyModificationUnits:   5,
-WarmStorageValueModificationUnits: 3,
+StorageKeyReadUnits:       5,
+StorageValueReadUnits:     2,
+StorageKeyAllocateUnits:   20,
+StorageValueAllocateUnits: 5,
+StorageKeyWriteUnits:      10,
+StorageValueWriteUnits:    3,
 ```
 
 #### Avoiding Complex Construction
@@ -355,12 +347,12 @@ price-sorted mempools are not particularly useful in high-throughput
 blockchains where the expected mempool size is ~0 or there is a bounded transaction
 lifetime (60 seconds by default on the `hypersdk`).
 
-#### Separate Metering for Storage Reads, Creations, Modifications
+#### Separate Metering for Storage Reads, Allocates, Writes
 To make the multidimensional fee implementation for the `hypersdk` simpler,
-it would have been possible to unify all storage operations (read, create,
-modify) into a single unit dimension. We opted not to go this route, however,
+it would have been possible to unify all storage operations (read, allocate,
+write) into a single unit dimension. We opted not to go this route, however,
 because `hypervm` designers often wish to regulate state growth much differently
-than state reads or state modification.
+than state reads or state writes.
 
 Fundamentally, it makes sense to combine resource usage into a single unit dimension
 if different operations are scaled substitutes of each other (an executor could translate
@@ -388,18 +380,6 @@ This constraint is equivalent to deciding whether to use a `uint8`, `uint16`, `u
 the estimate will be for a user to interact with state. Users are only charged, however,
 based on the amount of chunks actually read/written from/to state.
 
-#### Block-Based Storage Access Discounts
-If a state key has already been accessed in a given block, future access
-by the same transaction/future transactions will be more efficient for the
-`hypersdk` to handle because the corresponding state is already sitting in memory.
-The `hypersdk` automatically tracks which state has already been loaded from disk (whether
-read or modified) over an entire block and charges different fees accordingly.
-
-For example, if a transaction modifies a key and then another transaction
-is executed which modifies the same value, the net cost for modifying the key
-to the `hypervm` (and to the entire network) is much cheaper than modifying a
-new key.
-
 ### Nonce-less and Expiring Transactions
 `hypersdk` transactions don't use [nonces](https://help.myetherwallet.com/en/articles/5461509-what-is-a-nonce)
 to protect against replay attack like many other account-based blockchains. This means users

chain/block.go

@@ -94,7 +94,7 @@ func NewGenesisBlock(root ids.ID) *StatefulBlock {
 		// .../vms/proposervm/pre_fork_block.go#L201
 		Tmstmp: time.Date(2023, time.January, 1, 0, 0, 0, 0, time.UTC).UnixMilli(),
 
-		// StateRoot should include all allocations made when loading the genesis file
+		// StateRoot should include all allocates made when loading the genesis file
 		StateRoot: root,
 	}
 }

chain/builder.go

@@ -139,7 +139,6 @@ func BuildBlock(
 	// Batch fetch items from mempool to unblock incoming RPC/Gossip traffic
 	mempool.StartStreaming(ctx)
 	b.Txs = []*Transaction{}
-	usedKeys := set.NewSet[string](0) // prefetch map for transactions in block
 	for time.Since(start) < vm.GetTargetBuildDuration() {
 		prepareStreamLock.Lock()
 		txs := mempool.Stream(ctx, streamBatch)
@@ -318,8 +317,7 @@ func BuildBlock(
 				// Note, these calculations must match block verification exactly
 				// otherwise they will produce a different state root.
 				blockLock.RLock()
-				coldReads := make(map[string]uint16, len(stateKeys))
-				warmReads := make(map[string]uint16, len(stateKeys))
+				reads := make(map[string]uint16, len(stateKeys))
 				var invalidStateKeys bool
 				for k := range stateKeys {
 					v := storage[k]
@@ -328,11 +326,7 @@ func BuildBlock(
 						invalidStateKeys = true
 						break
 					}
-					if usedKeys.Contains(k) {
-						warmReads[k] = numChunks
-						continue
-					}
-					coldReads[k] = numChunks
+					reads[k] = numChunks
 				}
 				blockLock.RUnlock()
 				if invalidStateKeys {
@@ -345,8 +339,7 @@ func BuildBlock(
 					ctx,
 					feeManager,
 					authCUs,
-					coldReads,
-					warmReads,
+					reads,
 					sm,
 					r,
 					tsv,
@@ -388,7 +381,6 @@ func BuildBlock(
 				tsv.Commit()
 				b.Txs = append(b.Txs, tx)
 				results = append(results, result)
-				usedKeys.Add(stateKeys.List()...)
 				if tx.WarpMessage != nil {
 					if warpErr == nil {
 						// Add a bit if the warp message was verified

chain/dependencies.go

@@ -132,26 +132,21 @@ type Rules interface {
 	// Invariants:
 	// * Controllers must manage the max key length and max value length (max network
 	//   limit is ~2MB)
-	// * Cold key reads/modifications are assumed to be more expensive than warm
-	//   when estimating the max fee
+	// * Creating a new key involves first allocating and then writing
 	// * Keys are only charged once per transaction (even if used multiple times), it is
 	//   up to the controller to ensure multiple usage has some compute cost
 	//
 	// Interesting Scenarios:
 	// * If a key is created and then modified during a transaction, the second
-	//   read will be a warm read of 0 chunks (reads are based on disk contents before exec)
-	// * If a key is removed and then re-created during a transaction, it counts
-	//   as a modification and a creation instead of just a modification
-	GetColdStorageKeyReadUnits() uint64
-	GetColdStorageValueReadUnits() uint64 // per chunk
-	GetWarmStorageKeyReadUnits() uint64
-	GetWarmStorageValueReadUnits() uint64 // per chunk
-	GetStorageKeyCreateUnits() uint64
-	GetStorageValueCreateUnits() uint64 // per chunk
-	GetColdStorageKeyModificationUnits() uint64
-	GetColdStorageValueModificationUnits() uint64 // per chunk
-	GetWarmStorageKeyModificationUnits() uint64
-	GetWarmStorageValueModificationUnits() uint64 // per chunk
+	//   read will be a read of 0 chunks (reads are based on disk contents before exec)
+	// * If a key is removed and then re-created with the same value during a transaction,
+	//   it doesn't count as a modification (returning to the current value on-disk is a no-op)
+	GetStorageKeyReadUnits() uint64
+	GetStorageValueReadUnits() uint64 // per chunk
+	GetStorageKeyAllocateUnits() uint64
+	GetStorageValueAllocateUnits() uint64 // per chunk
+	GetStorageKeyWriteUnits() uint64
+	GetStorageValueWriteUnits() uint64 // per chunk
 
 	GetWarpConfig(sourceChainID ids.ID) (bool, uint64, uint64)
 

chain/fee_manager.go

@@ -15,11 +15,11 @@ import (
 )
 
 const (
-	Bandwidth           Dimension = 0
-	Compute             Dimension = 1
-	StorageRead         Dimension = 2
-	StorageCreate       Dimension = 3
-	StorageModification Dimension = 4
+	Bandwidth       Dimension = 0
+	Compute         Dimension = 1
+	StorageRead     Dimension = 2
+	StorageAllocate Dimension = 3
+	StorageWrite    Dimension = 4 // includes delete
 
 	FeeDimensions = 5