Set on-chain membership in a subnet when a reconfiguration is trigger…

…ed (filecoin-project#144) * use go-libipfs/files instead of ipfs/go-ipfs-files * fix drand test * Update .github/ISSUE_TEMPLATE/service_developer_bug_report.yml * Update .github/ISSUE_TEMPLATE/service_developer_bug_report.yml * Update service_developer_bug_report.yml * Update service_developer_bug_report.yml * Update service_developer_bug_report.yml * un-un-update filecoin-ffi * try to add statenetworkname method * add method implementation * add MPoolGetNonce * add two more methods: StateCall and StateDecodeParams * fix logging * deprecate MarketListRetrievalDeals, make gen * Fix up approximation and logging * run make gen * small doc patch on how to make gen after api changes * fix TestPrintGroupInfo * Eth API: make block parameter parsing sounder. * Eth API: fail when requesting future epochs. * add a test to verify block parameter soundness. * register rcmgr metrics * replace make all with make deps according to review * fix: EthAPI: use StateCompute for feeHistory; apply minimum gas premium (filecoin-project#10413) * cid key size * computing szPurge leads to deadlock and unneeded, remove * Review Response * Lint * lint * docsgen-cli * only register metrics once * lint * fix: eth API: return correct txIdx around null blocks (filecoin-project#10419) * fmt. * lint * feat:splitstore:Splitstore enabled by default (filecoin-project#10429) Discard mode --------- Co-authored-by: zenground0 <[email protected]> * Review Response * fix lint. * Configur buffer and threshold * lint * make gen * fix TestEthBlockHashesCorrect_MultiBlockTipset: skip null rounds. * switch to ChainAPI#ChainGetTipSetByHeight. * review comments * simplify test. * feat: chain: make fetching tipset by height 1000x faster * feat: mempool: Reduce minimum replace fee from 1.25x to 1.1x (filecoin-project#10416) However, we're leaving the default at 1.25x for backwards compatibility, for now. Also: 1. Actually use the configured replace fee ratio. 2. Store said ratios as percentages instead of floats. 1.25, or 1+1/(2^2), can be represented as a float. 1.1, or 1 + 1/(2 * 5), cannot. fixes filecoin-project#10415 * try to add statenetworkname method * add method implementation * add MPoolGetNonce * add two more methods: StateCall and StateDecodeParams * run make gen * Eth API: make block parameter parsing sounder. * Eth API: fail when requesting future epochs. * add a test to verify block parameter soundness. * fmt. * fix lint. * fix TestEthBlockHashesCorrect_MultiBlockTipset: skip null rounds. * switch to ChainAPI#ChainGetTipSetByHeight. * simplify test. * fix: EthAPI: use StateCompute for feeHistory; apply minimum gas premium (filecoin-project#10413) * fix: eth API: return correct txIdx around null blocks (filecoin-project#10419) * feat: mempool: Reduce minimum replace fee from 1.25x to 1.1x (filecoin-project#10416) However, we're leaving the default at 1.25x for backwards compatibility, for now. Also: 1. Actually use the configured replace fee ratio. 2. Store said ratios as percentages instead of floats. 1.25, or 1+1/(2^2), can be represented as a float. 1.1, or 1 + 1/(2 * 5), cannot. fixes filecoin-project#10415 * make gen * change log * fiix bakcport error filecoin-project#10427 * retract v1.20.2 * go mod tidy * make gen * feat: cli: Add an EVM command to fetch a contract's bytecode * feat: eth API: reject masked ID addresses embedded in f410f payloads We'll never get an actor/account deployed to one of these addresses (although we might get a placeholder). However, converting such an address to an f4 address is definitely wrong. * chore: update the FFI for release (filecoin-project#10435) Specifically, this updates the proofs libraries and enables CUDA by default. * feat: stmgr: skip tipset execution when possible * feat: eth: optimize receipt reading This optimizes the eth APIs (except the fee history one) to lookup the tipset state/receipts instead of computing the state. * feat: api: optimize ChainGetParentReceipts Read the receipts all at once instead of fetching them one-by-one. * test: eth: reduce chances of chain-reorgs affecting the test Now that this API is _much_ faster, we're more likely to "catch up" to the head faster than it can stabilize. I'm pretty sure the test was intended to be written this way anyways. * gateway: fix: drop overzealous guard on MsigGetVested * eth_feeHistory: migrate to using TipSetState. * Eth API: drop support for 'pending' block parameter. After transitioning from using StateCompute to loading receipts, we can no longer handle the 'pending' block without forcing computation. Eth Core Devs are evaluating a proposal to remove support on their end too. * Eth API: make net_version return the chain ID. * fix imports. * remove superfluous filter. * drop irrelevant test. * Eth API: make eth_getTransactionByBlock* ops return ErrUnsupported. * fix: itest: avoid failing the test when we race the miner I've been seeing quite a few tests failing here, so I'm hoping this will deflake them a bit. * feat: eth: Avoid StateCompute in EthTxnReceipt lookup (filecoin-project#10460) Instead: 1. Use the receipt from the message search. 2. Re-compute the gas fees that would have been charged. fixes filecoin-project#10418 Co-authored-by: raulk <[email protected]> * refactor: update cache to the new generic version (filecoin-project#10463) - Adds type safety. - Reduces allocations. - Fixes the drand cache (was storing by value, but retrieving by pointer) * fix: gateway: correctly apply the fee history lookback max This fix ensures that the full fee-history range falls into the allowable lookback rang. * Plumb through a proper Flush() method on all blockstores * feat: consensus: log ApplyBlock timing/gas stats * fix: stmgr: don't attempt to lookup genesis state (filecoin-project#10472) * fix: stmgr: don't attempt to lookup genesis state * stmgr: More correct fix for genesis state compute * Revert "Eth API: drop support for 'pending' block parameter." This reverts commit 9412753. * fix: build: drop drand incentinet servers * feat: gateway: export StateVerifierStatus * chore: update go-libp2p-pubsub to v0.9.2 * chore: update ffi to increase execution parallelism (filecoin-project#10480) This changes the default validation parallelism from 1 to 4, and makes it configurable with a LOTUS_FVM_CONCURRENCY environment variable. * fix connection gater integration test * chore: update go-libp2p to v0.26.2 * make mod-tidy ci happy * update go-libp2p-pubsub to v0.9.3 * update go-jsonrpc to v0.2.3 * ethrpc: Buffer sub messages if subscriber is slow * make lint happy * feat: apply gateway lookback limit to eth API lookback This change: 1. Introduces new "limited" API endpoints for EthGetTransactionByHash and EthGetTransactionReceipt that accept lookback-limits. 2. Implements the gateway version of these API endpoints by calling the limited variants with the default message search lookback limit. fixes filecoin-project#10412 * fix: actually apply limit * feat: expose limited apis on the gateway * fix: eth: handle a potential divide by zero in receipt handling This isn't really possible to hit on-chain at this piont (message won't be accepted) but we might as well be extra careful. fixes filecoin-project#10471 * Don't enforce walking receipt tree during compaction * Demote now common logs (filecoin-project#10516) Co-authored-by: zenground0 <[email protected]> * feat:config:force existing users to opt into new defaults (filecoin-project#10488) * Config default does not comment out EnableSplitstore * Loadability check * Remove test used for debugging * regexp for properly safe check that config is set * regexp for safely matching the EnableSpitstore field in the config * Add instructions for undeleting config and remind users to set splitstore false for full archive * UpdateConfig small docs and functional opts * make gen * Lint * Fix * nil pointer check on validate * Unit testing of EnableSplitstore cases * Address Review --------- Co-authored-by: zenground0 <[email protected]> * release v1.21.0-rc1 prep * chore: build: Update Changelog for v1.21.0 chore: build: Update Changelog for v1.21.0 * Add RPC API highlight Add RPC API highlight * Add callout to exchanges Add callout to exchanges about the execution traces. Also moving the `execution trace format changes` section to above the contributors section. * fix: miner: correctly count sector extensions * fix: miner: call ExtendSectorExpiration2 * chore: all: bump go-libipfs (cherry picked from commit f48c626) * Initialize with same length as partition Initialize the postParam.Partitions slice with the same length as i.Partitions before iterating over it in the loop. * build: docker: Update GO-version build: docker: Update GO-version * build: release: v1.21.0-rc2 build: release: v1.21.0-rc2 * Update go.mod Update go.mod * Update changelog for v1.21.0-rc2 Update changelog for v1.21.0-rc2 * perf: eth: gas estimate set applyTsMessages false (filecoin-project#10546) * have gas estimate call callInternal with applyTsMessages = false and other calls with applyTsMessages=true for gas caclulation optimization * set applyTsMessages = true in CallWithGas call in shed * update test with new callwithgas api optimization for eth call * Update chain/stmgr/call.go Co-authored-by: Łukasz Magiera <[email protected]> * env flag LOTUS_SKIP_APPLY_TS_MESSAGE_CALL_WITH_GAS must be 1 in order to have applyTsMessages change * env flag LOTUS_SKIP_APPLY_TS_MESSAGE_CALL_WITH_GAS must be 1 in order to have applyTsMessages change * make sure that even if we arent apply ts messages we grab ts messages from the particular user who is requesting gas estimation --------- Co-authored-by: Jiaying Wang <[email protected]> Co-authored-by: Łukasz Magiera <[email protected]> Co-authored-by: Ubuntu <[email protected]> * feat: supply: drop genesis market locked funds * feat: supply: only grab genesis msig locks for writes * Update Changelog with backports Update Changelog with backports * Bump version in changelog Bump version in changelog * Add longer validation time cause to changelog Add longer validation time cause to changelog * refactor: stop using deprecated io/ioutil * Fix error when injecting StateManager * new actor bundle, fix ci, upgrade fx * fix fx version * upgrade go version circle ci * make linter happy :) * more go version fixing * replace deprecated mac build * removing support for macos build * Fix ctrl-c for GOOD * bypass mpool rm for set membership --------- Co-authored-by: Marten Seemann <[email protected]> Co-authored-by: Jiaying Wang <[email protected]> Co-authored-by: Łukasz Magiera <[email protected]> Co-authored-by: Łukasz Magiera <[email protected]> Co-authored-by: Arsenii Petrovich <[email protected]> Co-authored-by: zenground0 <[email protected]> Co-authored-by: Arsenii Petrovich <[email protected]> Co-authored-by: Raúl Kripalani <[email protected]> Co-authored-by: ychiao <[email protected]> Co-authored-by: Aayush Rajasekaran <[email protected]> Co-authored-by: ZenGround0 <[email protected]> Co-authored-by: Travis Person <[email protected]> Co-authored-by: Travis Person <[email protected]> Co-authored-by: Steven Allen <[email protected]> Co-authored-by: Jennifer Wang <[email protected]> Co-authored-by: raulk <[email protected]> Co-authored-by: Peter Rabbitson <[email protected]> Co-authored-by: vyzo <[email protected]> Co-authored-by: Phi <[email protected]> Co-authored-by: Jorropo <[email protected]> Co-authored-by: Mikers <[email protected]> Co-authored-by: Ubuntu <[email protected]> Co-authored-by: Henrique Moniz <[email protected]> Co-authored-by: Denis Kolegov <[email protected]>
protofire · Apr 14, 2023 · 814d3cb · 814d3cb
1 parent 9d21b03
commit 814d3cb
Show file tree

Hide file tree

Showing 20 changed files with 489 additions and 199 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -213,14 +213,8 @@ jobs:
         description: gotestsum format. https://github.com/gotestyourself/gotestsum#format
       coverage:
         type: string
-        default: -coverprofile=coverage.txt -coverpkg=github.com/filecoin-project/lotus/...
+        default: "-coverprofile=/tmp/conformance.out -coverpkg ./chain/consensus/...,./chain/ipcagent/...,./chain/eudico-core/..."
         description: Coverage flag. Set to the empty string to disable.
-      codecov-upload:
-        type: boolean
-        default: true
-        description: |
-          Upload coverage report to https://codecov.io/. Requires the codecov API token to be
-          set as an environment variable for private projects.
     executor: << parameters.executor >>
     steps:
       - prepare
@@ -245,7 +239,6 @@ jobs:
           name: go test
           environment:
             TEST_RUSTPROOFS_LOGS: << parameters.proofs-log-test >>
-            SKIP_CONFORMANCE: "1"
             MIR_INTERCEPTOR_OUTPUT: "mir-interceptor"
             GOLOG_LOG_LEVEL: "info,mir-consensus=info,mir-manager=info"
           command: |
@@ -259,6 +252,7 @@ jobs:
               << parameters.coverage >> \
               << parameters.go-test-flags >> \
               << parameters.target >>
+            go tool cover -html=/tmp/conformance.out -o /tmp/test-artifacts/test-coverage.html
           no_output_timeout: 30m
       - run:
           command: |
@@ -272,13 +266,10 @@ jobs:
           path: /tmp/test-artifacts/<< parameters.suite >>.json
       - store_artifacts:
           path: /tmp/test-mir-interceptor
-      - when:
-          condition: << parameters.codecov-upload >>
-          steps:
-            - run:
-                shell: /bin/bash -eo pipefail
-                command: |
-                  bash <(curl -s https://codecov.io/bash)
+      - store_artifacts:
+          path: /tmp/test-artifacts/test-coverage.html
+      - store_artifacts:
+          path: /tmp/conformance.out
 
   test-mir-smoke:
     <<: *basic_test
@@ -372,10 +363,12 @@ workflows:
       - test-chain-unit:
           name: chain-unit-tests
           suite: consensus
+          go-test-flags: "-tags=spacenet -race -shuffle=on -v -count=1"
           target: "./chain/consensus/..."
       - test-ipcagent-unit:
           name: ipcagent-unit-tests
           suite: consensus
+          go-test-flags: "-tags=spacenet -race -shuffle=on -v -count=1"
           target: "./chain/ipcagent/..."
       - test-mir-smoke:
           name: mir-smoke-tests
@@ -418,15 +411,15 @@ workflows:
       - nightly-test-mir-with-mangler:
           name: nightly-mir-mangler-tests
           suite: consensus
-          go-test-flags: "-tags=spacenet -count=1 -v -timeout 120m -run TestMirWithMangler"
+          go-test-flags: "-tags=spacenet -count=1 -v -timeout=120m -run TestMirWithMangler"
           target: "./itests/mir_test.go"
       - nightly-test-mir-basic:
           name: nightly-mir-basic-tests
           suite: consensus
-          go-test-flags: "-tags=spacenet -count=1 -v -timeout 20m -run TestMirBasic"
+          go-test-flags: "-tags=spacenet -count=1 -v -timeout=20m -run TestMirBasic"
           target: "./itests/mir_test.go"
       - nightly-test-mir-reconfiguration:
           name: nightly-mir-reconfiguration-tests
           suite: consensus
-          go-test-flags: "-tags=spacenet -count=1 -v -timeout 20m -run TestMirReconfiguration"
+          go-test-flags: "-tags=spacenet -count=1 -v -timeout=20m -run TestMirReconfiguration"
           target: "./itests/mir_test.go"
diff --git a/.golangci.yml b/.golangci.yml
@@ -91,6 +91,11 @@ issues:
       - staticcheck
       text: "SA1019: \"github.com/libp2p/go-libp2p-core"
 
+    - path: itests/wdpost_dispute_test.go
+      text: "cannot use client"
+      linters:
+        - typecheck
+
 linters-settings:
   goconst:
     min-occurrences: 6
diff --git a/chain/consensus/common.go b/chain/consensus/common.go
@@ -25,6 +25,7 @@ import (
 	bstore "github.com/filecoin-project/lotus/blockstore"
 	"github.com/filecoin-project/lotus/build"
 	"github.com/filecoin-project/lotus/chain/actors/builtin"
+	"github.com/filecoin-project/lotus/chain/consensus/mir/membership"
 	"github.com/filecoin-project/lotus/chain/state"
 	"github.com/filecoin-project/lotus/chain/stmgr"
 	"github.com/filecoin-project/lotus/chain/store"
@@ -38,6 +39,8 @@ import (
 // Common operations shared by all consensus algorithm implementations.
 var log = logging.Logger("consensus-common")
 
+var DefaultGatewayAddr, _ = address.NewIDAddress(64)
+
 // RunAsyncChecks accepts a list of checks to perform in parallel.
 //
 // Each consensus algorithm may choose to perform a set of different
@@ -264,7 +267,6 @@ func checkBlockMessages(ctx context.Context, sm *stmgr.StateManager, cs *store.C
 		if err != nil {
 			return xerrors.Errorf("failed to store message %s: %w", m.Cid(), err)
 		}
-
 		k := cbg.CborCid(c)
 		if err := bmArr.Set(uint64(i), &k); err != nil {
 			return xerrors.Errorf("failed to put bls message at index %d: %w", i, err)
@@ -277,19 +279,23 @@ func checkBlockMessages(ctx context.Context, sm *stmgr.StateManager, cs *store.C
 			return xerrors.Errorf("block had invalid signed message at index %d: %w", i, err)
 		}
 
-		if err := checkMsg(m); err != nil {
-			return xerrors.Errorf("block had invalid secpk message at index %d: %w", i, err)
-		}
+		// if this is a config message no need to check the message
+		if !membership.IsConfigMsg(DefaultGatewayAddr, &m.Message) {
 
-		// `From` being an account actor is only validated inside the `vm.ResolveToDeterministicAddr` call
-		// in `StateManager.ResolveToDeterministicAddress` here (and not in `checkMsg`).
-		kaddr, err := sm.ResolveToDeterministicAddress(ctx, m.Message.From, baseTs)
-		if err != nil {
-			return xerrors.Errorf("failed to resolve key addr: %w", err)
-		}
+			if err := checkMsg(m); err != nil {
+				return xerrors.Errorf("block had invalid secpk message at index %d: %w", i, err)
+			}
 
-		if err := AuthenticateMessage(m, kaddr); err != nil {
-			return xerrors.Errorf("failed to validate signature: %w", err)
+			// `From` being an account actor is only validated inside the `vm.ResolveToDeterministicAddr` call
+			// in `StateManager.ResolveToDeterministicAddress` here (and not in `checkMsg`).
+			kaddr, err := sm.ResolveToDeterministicAddress(ctx, m.Message.From, baseTs)
+			if err != nil {
+				return xerrors.Errorf("failed to resolve key addr: %w", err)
+			}
+
+			if err := AuthenticateMessage(m, kaddr); err != nil {
+				return xerrors.Errorf("failed to validate signature: %w", err)
+			}
 		}
 
 		c, err := store.PutMessage(ctx, tmpbs, m)

diff --git a/chain/consensus/compute_state.go b/chain/consensus/compute_state.go
@@ -30,6 +30,7 @@ import (
 	"github.com/filecoin-project/lotus/chain/actors/builtin"
 	"github.com/filecoin-project/lotus/chain/actors/builtin/cron"
 	"github.com/filecoin-project/lotus/chain/actors/builtin/reward"
+	"github.com/filecoin-project/lotus/chain/consensus/mir/membership"
 	"github.com/filecoin-project/lotus/chain/rand"
 	"github.com/filecoin-project/lotus/chain/stmgr"
 	"github.com/filecoin-project/lotus/chain/store"
@@ -214,6 +215,31 @@ func (t *TipSetExecutor) ApplyBlocks(ctx context.Context,
 			if _, found := processedMsgs[m.Cid()]; found {
 				continue
 			}
+
+			// Check if the block includes a config message to set the
+			// new membership and execute it implicitly.
+			// FIXME: Setting default gateway address here, this should
+			// maybe change
+			if membership.IsConfigMsg(DefaultGatewayAddr, m) {
+				r, err := vmi.ApplyImplicitMessage(ctx, m) // nolint
+				if err != nil {
+					return cid.Undef, cid.Undef, xerrors.Errorf("running cron: %w", err)
+				}
+
+				if em != nil {
+					if err := em.MessageApplied(ctx, ts, cm.Cid(), m, r, true); err != nil {
+						return cid.Undef, cid.Undef, xerrors.Errorf("callback failed on set-membership: %w", err)
+					}
+				}
+
+				if r.ExitCode != 0 {
+					return cid.Undef, cid.Undef, xerrors.Errorf("membership exit was non-zero: %d", r.ExitCode)
+				}
+				processedMsgs[m.Cid()] = struct{}{}
+				// if config message executed we can move to the next one
+				continue
+			}
+
 			r, err := vmi.ApplyMessage(ctx, cm)
 			if err != nil {
 				return cid.Undef, cid.Undef, err

diff --git a/chain/consensus/mir/cbor_gen.go b/chain/consensus/mir/cbor_gen.go
diff --git a/chain/consensus/mir/conf_manager.go b/chain/consensus/mir/conf_manager.go
@@ -14,8 +14,10 @@ import (
 	"github.com/filecoin-project/mir/pkg/client"
 	mirproto "github.com/filecoin-project/mir/pkg/pb/requestpb"
 	t "github.com/filecoin-project/mir/pkg/types"
+	"github.com/filecoin-project/mir/pkg/util/maputil"
 
 	"github.com/filecoin-project/lotus/chain/consensus/mir/db"
+	"github.com/filecoin-project/lotus/chain/consensus/mir/membership"
 )
 
 const (
@@ -29,27 +31,45 @@ var (
 	// NextAppliedConfigurationNumberKey is used to store AppliedConfigurationNumber
 	// that is the maximum configuration request number that has been applied.
 	NextAppliedConfigurationNumberKey = datastore.NewKey("mir/next-applied-config-number")
-	// ReconfigurationVotesKey is used to store configuration votes.
-	ReconfigurationVotesKey = datastore.NewKey("mir/reconfiguration-votes")
+	// ConfigurationVotesKey is used to store configuration votes.
+	ConfigurationVotesKey = datastore.NewKey("mir/reconfiguration-votes")
 )
 
 var _ client.Client = &ConfigurationManager{}
 
 type ConfigurationManager struct {
-	ctx           context.Context // Parent context
-	ds            db.DB           // Persistent storage.
-	id            string          // Manager ID.
-	nextReqNo     uint64          // The number that will be used in the next configuration Mir request.
-	nextAppliedNo uint64          // The number of the next configuration Mir request that will be applied.
+	ctx                  context.Context // Parent context
+	ds                   db.DB           // Persistent storage.
+	id                   string          // Manager ID.
+	nextReqNo            uint64          // The number that will be used in the next configuration Mir request.
+	nextAppliedNo        uint64          // The number of the next configuration Mir request that will be applied.
+	initialConfiguration membership.Info // Initial membership information.
 }
 
 func NewConfigurationManager(ctx context.Context, ds db.DB, id string) (*ConfigurationManager, error) {
 	cm := &ConfigurationManager{
-		ctx:           ctx,
-		ds:            ds,
-		id:            id,
-		nextReqNo:     0,
-		nextAppliedNo: 0,
+		ctx:                  ctx,
+		ds:                   ds,
+		id:                   id,
+		nextReqNo:            0,
+		nextAppliedNo:        0,
+		initialConfiguration: membership.Info{},
+	}
+	err := cm.recover()
+	if err != nil {
+		return nil, err
+	}
+	return cm, nil
+}
+
+func NewConfigurationManagerWithMembershipInfo(ctx context.Context, ds db.DB, id string, info *membership.Info) (*ConfigurationManager, error) {
+	cm := &ConfigurationManager{
+		ctx:                  ctx,
+		ds:                   ds,
+		id:                   id,
+		nextReqNo:            0,
+		nextAppliedNo:        0,
+		initialConfiguration: *info,
 	}
 	err := cm.recover()
 	if err != nil {
@@ -86,6 +106,10 @@ func (cm *ConfigurationManager) NewTX(_ uint64, data []byte) (*mirproto.Request,
 	return &r, nil
 }
 
+func (cm *ConfigurationManager) GetInitialMembershipInfo() membership.Info {
+	return cm.initialConfiguration
+}
+
 // Done marks a configuration request as done. It will no longer be among the request returned by Pending.
 func (cm *ConfigurationManager) Done(txNo t.ReqNo) error {
 	cm.nextAppliedNo = uint64(txNo) + 1
@@ -204,7 +228,7 @@ func (cm *ConfigurationManager) getAppliedConfigurationNumber() uint64 {
 
 func (cm *ConfigurationManager) GetConfigurationVotes() map[uint64]map[string]map[t.NodeID]struct{} {
 	votes := make(map[uint64]map[string]map[t.NodeID]struct{})
-	b, err := cm.ds.Get(cm.ctx, ReconfigurationVotesKey)
+	b, err := cm.ds.Get(cm.ctx, ConfigurationVotesKey)
 	if errors.Is(err, datastore.ErrNotFound) {
 		log.With("validator", cm.id).Info("stored reconfiguration votes not found")
 		return votes
@@ -225,7 +249,7 @@ func (cm *ConfigurationManager) GetConfigurationVotes() map[uint64]map[string]ma
 }
 
 func (cm *ConfigurationManager) StoreConfigurationVotes(votes map[uint64]map[string]map[t.NodeID]struct{}) error {
-	recs := storeConfigurationVotes(votes)
+	recs := StoreConfigurationVotes(votes)
 	r := VoteRecords{
 		Records: recs,
 	}
@@ -234,7 +258,7 @@ func (cm *ConfigurationManager) StoreConfigurationVotes(votes map[uint64]map[str
 	if err := r.MarshalCBOR(b); err != nil {
 		return err
 	}
-	if err := cm.ds.Put(cm.ctx, ReconfigurationVotesKey, b.Bytes()); err != nil {
+	if err := cm.ds.Put(cm.ctx, ConfigurationVotesKey, b.Bytes()); err != nil {
 		log.With("validator", cm.id).Warnf("failed to put reconfiguration votes: %v", err)
 	}
 
@@ -269,15 +293,18 @@ func GetConfigurationVotes(vr []VoteRecord) map[uint64]map[string]map[t.NodeID]s
 	return m
 }
 
-func storeConfigurationVotes(votes map[uint64]map[string]map[t.NodeID]struct{}) []VoteRecord {
+func StoreConfigurationVotes(votes map[uint64]map[string]map[t.NodeID]struct{}) []VoteRecord {
 	var vs []VoteRecord
-	for n, hashToValidatorsVotes := range votes {
-		for h, nodeIDs := range hashToValidatorsVotes {
+
+	for _, n := range maputil.GetSortedKeys(votes) {
+		hashToValidatorsVotes := votes[n]
+		for _, h := range maputil.GetSortedKeys(hashToValidatorsVotes) {
+			nodeIDs := hashToValidatorsVotes[h]
 			e := VoteRecord{
 				ConfigurationNumber: n,
 				ValSetHash:          h,
 			}
-			for n := range nodeIDs {
+			for _, n := range maputil.GetSortedKeys(nodeIDs) {
 				e.VotedValidators = append(e.VotedValidators, VotedValidator{n.Pb()})
 			}
 			vs = append(vs, e)