chain-maind failing to start with error: chain-maind[1953]: panic: Value missing for hash 09b5abfdd53c9458c3bfa2c8d15be338ad011dedbea285ee3b82ff9a9e32ea62 corresponding to nodeKey 6e09b5abfdd53c9458c3bfa2c8d15be338ad011dedbea285ee3b82ff9a9e32ea62

[raulillo82]

Good morning.

My node fails to start. Due to some btrfs error that I was not able to debug so far, root filesystem went to read-only. After restarting the node and running the unjail transaction successfully, node won't start again. When restarting the service manually, I can see the error attached at the end of this description, main problem is attached to the issue description as well.

Some information about my node:

Node has 4 CPU, 100GB SSD, 32GB of RAM, ethernet gigabit and a fiber 1GB symmetric connection to the internet. Half of the space is free, and as said, it's running on a btrfs filesystem (Opensuse Leap 15.2).

I did send an unjail transaction that was (apparently) successful.

raul@cro:~> sudo cat /etc/systemd/system/chain-listener.service
[Unit]
Description=Crypto.com chain-maind listener
ConditionPathExists=/home/raul/croeseid/chain-maind
After=network.target

[Service]
Type=simple
Restart=on-failure
RestartSec=10
User=raul
LimitNOFILE=4096
Environment="RUST_LOG=info"
WorkingDirectory=/home/raul/croeseid
ExecStart=/home/raul/croeseid/chain-maind start

[Install]
WantedBy=multi-user.target

raul@cro:~> sudo systemctl restart chain-listener.service && sudo journalctl -u chain-listener.service -f
dic 15 11:41:15 cro systemd[1]: Stopped Crypto.com chain-maind listener.
dic 15 11:41:15 cro systemd[1]: Started Crypto.com chain-maind listener.
dic 15 11:41:15 cro chain-maind[2253]: I[2020-12-15|11:41:15.589] starting ABCI with Tendermint                module=main
dic 15 11:41:16 cro chain-maind[2253]: panic: Value missing for hash 09b5abfdd53c9458c3bfa2c8d15be338ad011dedbea285ee3b82ff9a9e32ea62 corresponding to nodeKey 6e09b5abfdd53c9458c3bfa2c8d15be338ad011dedbea285ee3b82ff9a9e32ea62
dic 15 11:41:16 cro chain-maind[2253]: goroutine 1 [running]:
dic 15 11:41:16 cro chain-maind[2253]: github.com/cosmos/iavl.(*nodeDB).GetNode(0xc000f028a0, 0xc000fae5c0, 0x20, 0x20, 0x0)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/cosmos/[email protected]/nodedb.go:90 +0x6ed
dic 15 11:41:16 cro chain-maind[2253]: github.com/cosmos/iavl.(*Node).getLeftNode(...)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/cosmos/[email protected]/node.go:421
dic 15 11:41:16 cro chain-maind[2253]: github.com/cosmos/iavl.(*Node).calcHeightAndSize(0xc001ea52c0, 0xc001ebf760)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/cosmos/[email protected]/node.go:433 +0x198
dic 15 11:41:16 cro chain-maind[2253]: github.com/cosmos/iavl.(*MutableTree).recursiveSet(0xc0011b5a40, 0xc00142e780, 0xc003b7b4c0, 0x1d, 0x20, 0xc0055d7c74, 0x2, 0x2, 0xc0018b4a00, 0x0, ...)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:176 +0x345
dic 15 11:41:16 cro chain-maind[2253]: github.com/cosmos/iavl.(*MutableTree).recursiveSet(0xc0011b5a40, 0xc00142e6e0, 0xc003b7b4c0, 0x1d, 0x20, 0xc0055d7c74, 0x2, 0x2, 0xc0018b4a00, 0x0, ...)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:166 +0x2be
dic 15 11:41:16 cro chain-maind[2253]: github.com/cosmos/iavl.(*MutableTree).recursiveSet(0xc0011b5a40, 0xc0017fd540, 0xc003b7b4c0, 0x1d, 0x20, 0xc0055d7c74, 0x2, 0x2, 0xc0018b4a00, 0x0, ...)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:169 +0x488
dic 15 11:41:16 cro chain-maind[2253]: github.com/cosmos/iavl.(*MutableTree).recursiveSet(0xc0011b5a40, 0xc0017fd4a0, 0xc003b7b4c0, 0x1d, 0x20, 0xc0055d7c74, 0x2, 0x2, 0xc0018b4a00, 0x0, ...)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:169 +0x488
dic 15 11:41:16 cro chain-maind[2253]: github.com/cosmos/iavl.(*MutableTree).recursiveSet(0xc0011b5a40, 0xc0017fd400, 0xc003b7b4c0, 0x1d, 0x20, 0xc0055d7c74, 0x2, 0x2, 0xc0018b4a00, 0x0, ...)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:166 +0x2be
dic 15 11:41:16 cro chain-maind[2253]: github.com/cosmos/iavl.(*MutableTree).recursiveSet(0xc0011b5a40, 0xc0017fd360, 0xc003b7b4c0, 0x1d, 0x20, 0xc0055d7c74, 0x2, 0x2, 0xc0018b4a00, 0x0, ...)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:169 +0x488
dic 15 11:41:16 cro chain-maind[2253]: github.com/cosmos/iavl.(*MutableTree).recursiveSet(0xc0011b5a40, 0xc005344320, 0xc003b7b4c0, 0x1d, 0x20, 0xc0055d7c74, 0x2, 0x2, 0xc0018b4a00, 0x0, ...)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:169 +0x488
dic 15 11:41:16 cro chain-maind[2253]: github.com/cosmos/iavl.(*MutableTree).recursiveSet(0xc0011b5a40, 0xc0037635e0, 0xc003b7b4c0, 0x1d, 0x20, 0xc0055d7c74, 0x2, 0x2, 0xc0018b4a00, 0x0, ...)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:166 +0x2be
dic 15 11:41:16 cro chain-maind[2253]: github.com/cosmos/iavl.(*MutableTree).recursiveSet(0xc0011b5a40, 0xc003763540, 0xc003b7b4c0, 0x1d, 0x20, 0xc0055d7c74, 0x2, 0x2, 0xc0018b4a00, 0x0, ...)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:169 +0x488
dic 15 11:41:16 cro chain-maind[2253]: github.com/cosmos/iavl.(*MutableTree).recursiveSet(0xc0011b5a40, 0xc0037634a0, 0xc003b7b4c0, 0x1d, 0x20, 0xc0055d7c74, 0x2, 0x2, 0xc0018b4a00, 0x0, ...)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:169 +0x488
dic 15 11:41:16 cro chain-maind[2253]: github.com/cosmos/iavl.(*MutableTree).recursiveSet(0xc0011b5a40, 0xc001ea41e0, 0xc003b7b4c0, 0x1d, 0x20, 0xc0055d7c74, 0x2, 0x2, 0xc0018b4a00, 0xc001ea0270, ...)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:169 +0x488
dic 15 11:41:16 cro chain-maind[2253]: github.com/cosmos/iavl.(*MutableTree).set(0xc0011b5a40, 0xc003b7b4c0, 0x1d, 0x20, 0xc0055d7c74, 0x2, 0x2, 0xc001ea4be0, 0xf, 0x14, ...)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:128 +0x137
dic 15 11:41:16 cro chain-maind[2253]: github.com/cosmos/iavl.(*MutableTree).Set(0xc0011b5a40, 0xc003b7b4c0, 0x1d, 0x20, 0xc0055d7c74, 0x2, 0x2, 0x1b345a8)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:100 +0x7a
dic 15 11:41:16 cro chain-maind[2253]: github.com/cosmos/cosmos-sdk/store/iavl.(*Store).Set(0xc0015be280, 0xc003b7b4c0, 0x1d, 0x20, 0xc0055d7c74, 0x2, 0x2)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/crypto-com/[email protected]/store/iavl/store.go:165 +0x82
dic 15 11:41:16 cro chain-maind[2253]: github.com/cosmos/cosmos-sdk/store/cache.(*CommitKVStoreCache).Set(0xc001ebf840, 0xc003b7b4c0, 0x1d, 0x20, 0xc0055d7c74, 0x2, 0x2)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/crypto-com/[email protected]/store/cache/cache.go:124 +0x13a
dic 15 11:41:16 cro chain-maind[2253]: github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Write(0xc001459020)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/crypto-com/[email protected]/store/cachekv/store.go:128 +0x610
dic 15 11:41:16 cro chain-maind[2253]: github.com/cosmos/cosmos-sdk/store/cachemulti.Store.Write(0x1d015a0, 0xc001458b70, 0xc001458ba0, 0xc0015c4150, 0x0, 0x0, 0x0)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/crypto-com/[email protected]/store/cachemulti/store.go:114 +0xaa
dic 15 11:41:16 cro chain-maind[2253]: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Commit(0xc000103980, 0x0, 0x0, 0x0, 0x0)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/crypto-com/[email protected]/baseapp/abci.go:302 +0x282
dic 15 11:41:16 cro chain-maind[2253]: github.com/tendermint/tendermint/abci/client.(*localClient).CommitSync(0xc000f03bc0, 0x0, 0x0, 0x0)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/tendermint/[email protected]/abci/client/local_client.go:258 +0xab
dic 15 11:41:16 cro chain-maind[2253]: github.com/tendermint/tendermint/proxy.(*appConnConsensus).CommitSync(0xc000b7fea0, 0x0, 0x0, 0xc001431b00)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/tendermint/[email protected]/proxy/app_conn.go:93 +0x33
dic 15 11:41:16 cro chain-maind[2253]: github.com/tendermint/tendermint/state.(*BlockExecutor).Commit(0xc000ded180, 0xb, 0x0, 0xc002852d70, 0x6, 0xc002891900, 0x12, 0x1, 0xc675c, 0xc000044bc0, ...)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/tendermint/[email protected]/state/execution.go:225 +0x23f
dic 15 11:41:16 cro chain-maind[2253]: github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc000ded180, 0xb, 0x0, 0xc002852d70, 0x6, 0xc002891900, 0x12, 0x1, 0xc675c, 0xc000044bc0, ...)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/tendermint/[email protected]/state/execution.go:177 +0x7c5
dic 15 11:41:16 cro chain-maind[2253]: github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlock(0xc0018b6748, 0xb, 0x0, 0xc002852d70, 0x6, 0xc002891900, 0x12, 0x1, 0xc675b, 0xc002891920, ...)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/tendermint/[email protected]/consensus/replay.go:503 +0x268
dic 15 11:41:16 cro chain-maind[2253]: github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(0xc00383c748, 0xb, 0x0, 0xc002852d70, 0x6, 0xc002891900, 0x12, 0x1, 0xc675b, 0xc002891920, ...)
dic 15 11:41:16 cro chain-maind[2253]:         /go/pkg/mod/github.com/tendermint/[email protected]/consensus/replay.go:416 +0xd18
dic 15 11:41:16 cro chain-maind[2253]: github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc0018b6748, 0x1d06140, 0xc000212f70, 0xc001373290, 0x203000)
dic 15 11:41:16 cro systemd[1]: chain-listener.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
dic 15 11:41:16 cro systemd[1]: chain-listener.service: Unit entered failed state.
dic 15 11:41:16 cro systemd[1]: chain-listener.service: Failed with result 'exit-code'.
dic 15 11:41:17 cro systemd[1]: chain-listener.service: Service RestartSec=1s expired, scheduling restart.
dic 15 11:41:17 cro systemd[1]: Stopped Crypto.com chain-maind listener.
dic 15 11:41:17 cro systemd[1]: chain-listener.service: Start request repeated too quickly.
dic 15 11:41:17 cro systemd[1]: Failed to start Crypto.com chain-maind listener.
dic 15 11:41:17 cro systemd[1]: chain-listener.service: Unit entered failed state.
dic 15 11:41:17 cro systemd[1]: chain-listener.service: Failed with result 'exit-code'.

Help will be much appreciated, I'm also available in the discord channel or gitter.

[raulillo82]

I managed to work around this by doing the following. Please take into account that this node had been running the previous testnet, thaler, so don't be surprised if you see old configurations or missing new configurations. I did apply some of the changes that I found in newer documentations.

• Make sure services are stopped. Stop them otherwise (please take into account that the service might have a different name for you):
  sudo systemctl stop chain-listener.service

• Apply some missing new configurations, I suppose the last one regarding "state-sync" is specially important, as it will allow to start the node quite faster than by downloading the whole chain again (sorry if the terms are not 100% accurate). The steps are here:
  https://chain.crypto.com/docs/getting-started/croeseid-testnet.html#step-2-configure-chain-maind (only section 2.2 to be followed).

• Rename the data directory inside of your .chain_maind directory. In my case, it was under ~/chain_maind but it might be different in other cases:
  mv ~/.chain_maind/data ~/.chain_maind/data.old

• Retrieve the file priv_validator_state.json from the old data:
  cp ~/.chain_maind/data.old/priv_validator_state.json ~/.chain_maind/data

• Restart services. Please take into account that the service name might be different for you as well. Adapt the command accordingly, or even using tab-completion in case of doubts:
  sudo systemctl restart chain-listener.service

• Check logs, you can see which block is being signed on them (again, the service name might be different):
  sudo journalctl -u chain-listener.service -f

• You can compare the last height (block number) in either any of the explorers or by running this command:
  curl -s https://testnet-croeseid-1.crypto.com:26657/commit | jq "{height: .result.signed_header.header.height}"

• Unjail the node, remember that this will only be possible after the unjail period has expired. The path for the command might be different for your node (look for the file chain-maind if it can't be found). The "--from" parameter has to be adapted with your node name, change XXX-node for your real node name, if the name is ghostbusters, then it would look like: "--from=ghostbusters"
  ~/croeseid/chain-maind tx slashing unjail --from=XXX-node --chain-id testnet-croeseid-1 --gas 200000 --gas-prices 0.1basetcro --node https://testnet-croeseid-1.crypto.com:26657

• Within the time, whenever it's checked that everything is working fine, the "data.old" directory can be removed. That will save a lot of space in case of running out of disk if the "state-sync" option has been enabled (remember this is only possible for validator nodes, not for full nodes).

I will leave the issue open just in case any developer want to look further into it, but otherwise feel free to close it.