[v6.0.2] Error during startup

[muratso]

Summary of Bug

After upgrading my node I started seeing the error below. I haven't done anything special to get this error.

10:59PM INF starting ABCI with Tendermint
10:59PM ERR WARNING: The minimum-gas-prices config in app.toml is set to the empty string. This defaults to 0 in the current version, but will error in the next version (SDK v0.45). Please explicitly put the desired minimum-gas-prices in your app.toml.
10:59PM INF Starting multiAppConn service impl=multiAppConn module=proxy
10:59PM INF Starting localClient service connection=query impl=localClient module=abci-client
10:59PM INF Starting localClient service connection=snapshot impl=localClient module=abci-client
10:59PM INF Starting localClient service connection=mempool impl=localClient module=abci-client
10:59PM INF Starting localClient service connection=consensus impl=localClient module=abci-client
10:59PM INF Starting EventBus service impl=EventBus module=events
10:59PM INF Starting PubSub service impl=PubSub module=pubsub
10:59PM INF Starting IndexerService service impl=IndexerService module=txindex
10:59PM INF ABCI Handshake App Info hash="R��g�f���qFw��Vm�(��,�\x01\x01�A\\DA<\x1c\x14" height=9464280 module=consensus protocol-version=0 software-version=v6.0.2
10:59PM INF ABCI Replay Blocks appHeight=9464280 module=consensus stateHeight=9464280 storeHeight=9464281
10:59PM INF Replay last block using real app module=consensus
Error: error during handshake: error on replay: wrong Block.Header.LastResultsHash.  Expected 26DEDC530232A503B5466897D172075BC06F38D5D81C63D417F3EE669BB26E82, got 46C8FD6CF82DE70A08376F45F8F73EE18793AAA4C46F13602962CDBC3836338D

Version

v6.0.2

Steps to Reproduce

Simply upgraded the node from 6.0.0 to 6.0.2

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[glnro]

Hi @muratso the official release version for mainnet is still Vega v6.0.0

[muratso]

Since the v6.0.2 had been officially released and tagged as "latest", I assumed it was safe to use it. In this case, I'm going to restore from a backup and roll back my node to v6.0.0 :). Thanks for the info.

[glnro]

@muratso Just going to keep this open to continue investigating

[okwme]

hi @muratso you're correct it is assumed that v6.0.2 is correct and can be run.
we swapped out v6.0.0 with v6.0.2 and have not encountered any errors.
is there any other contextual information you can provide that might help us debug this further?

[okwme]

also happy to connect in a more synchronous channel than github if you have an email, telegram or discord handle

[muratso]

🤔 Let me give it a try in one thing. If it's safe to use it, let's assume something got corrupted during the node shutdown when I was upgrading it. I'll try to restore the disk from a snapshot and retry the upgrade.
If it works I'll come back here and let you guys know 🙂.

[tac0turtle]

@muratso hey, thanks for pointing this out. Could you post your pruning setting for your node?

[okwme]

thank you @muratso !

[okwme]

Hi @muratso we've also now detected nondeterminism. Please use v6.0.0 until we find the source of the breaking change.

[muratso]

@marbar3778 just answering your question while I restore the disk. We haven't tuned the prune options, so I believe we're using the default for pretty much all the options related to pruning.

pruning = "default"
pruning-keep-recent = "0"
pruning-keep-every = "0"
pruning-interval = "0"
min-retain-blocks = 0
snapshot-interval = 0

[muratso]

@okwme cool. Thanks for letting me know! Would you mind pinging me on Telegram quickly?

[glnro]

This is the consensus failure I encountered:

11:06AM INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"D844B9436D1AEBB4940F4777EB213FFE0B16BA3C9D9E47BB1BE6FA8BA28B9C6B","parts":{"hash":"3F5ACFC2C93756A89994AE6002544099CC7661CC41A7C284EC70D9E39E0666BD","total":1}},"height":9470262,"pol_round":-1,"round":0,"signature":"vs3N8lxiWP3GizGvwz6VlSVRJwU4nnDKqj+Tt9n9ENaL3LPBrC9D9NRsXGiDy6543KtVQEjKYnZtIMtGenTGBw==","timestamp":"2022-02-18T11:06:15.897776607Z"}
11:06AM INF received complete proposal block hash=D844B9436D1AEBB4940F4777EB213FFE0B16BA3C9D9E47BB1BE6FA8BA28B9C6B height=9470262 module=consensus
11:06AM ERR prevote step: ProposalBlock is invalid err="wrong Block.Header.LastResultsHash.  Expected 23D468B89B1CDAF45C0A14807A7C9403C56815EBE0AFAB4B3E10B0B97ACD6E63, got 0274EE0686AAFFEACF874F9CE3AECC8EB510AF36BFAFF2F34E0F909A166A3268" height=9470262 module=consensus round=0
11:06AM ERR CONSENSUS FAILURE!!! err="precommit step; +2/3 prevoted for an invalid block: wrong Block.Header.LastResultsHash.  Expected 23D468B89B1CDAF45C0A14807A7C9403C56815EBE0AFAB4B3E10B0B97ACD6E63, got 0274EE0686AAFFEACF874F9CE3AECC8EB510AF36BFAFF2F34E0F909A166A3268" module=consensus stack="goroutine 1702 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:24 +0x65\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine.func2()\n\tgithub.com/tendermint/[email protected]/consensus/state.go:726 +0x4c\npanic({0x18a53c0, 0xc019c67e60})\n\truntime/panic.go:1038 +0x215\ngithub.com/tendermint/tendermint/consensus.(*State).enterPrecommit(0xc000ebd500, 0x908136, 0x0)\n\tgithub.com/tendermint/[email protected]/consensus/state.go:1390 +0x171f\ngithub.com/tendermint/tendermint/consensus.(*State).addVote(0xc000ebd500, 0xc019c1b9a0, {0xc0084f5380, 0x28})\n\tgithub.com/tendermint/[email protected]/consensus/state.go:2104 +0x188f\ngithub.com/tendermint/tendermint/consensus.(*State).tryAddVote(0xc000ebd500, 0xc019c1b9a0, {0xc0084f5380, 0xc019c15600})\n\tgithub.com/tendermint/[email protected]/consensus/state.go:1930 +0x2c\ngithub.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc000ebd500, {{0x1fdf4e0, 0xc019c40748}, {0xc0084f5380, 0x0}})\n\tgithub.com/tendermint/[email protected]/consensus/state.go:838 +0x40b\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0xc000ebd500, 0x0)\n\tgithub.com/tendermint/[email protected]/consensus/state.go:762 +0x419\ncreated by github.com/tendermint/tendermint/consensus.(*State).OnStart\n\tgithub.com/tendermint/[email protected]/consensus/state.go:378 +0x12f\n"
11:06AM INF Stopping baseWAL service impl={"Logger":{}} module=consensus wal=/root/.gaia/data/cs.wal/wal
11:06AM INF Stopping Group service impl={"Dir":"/root/.gaia/data/cs.wal","Head":{"ID":"V4HwICDo2v8Y:/root/.gaia/data/cs.wal/wal","Path":"/root/.gaia/data/cs.wal/wal"},"ID":"group:V4HwICDo2v8Y:/root/.gaia/data/cs.wal/wal","Logger":{}} module=consensus wal=/root/.gaia/data/cs.wal/wal

[yaruwangway]

tested locally is fine:
tested locally a new network with node of v6.0.0 and v6.0.2, tried a bunch of tx including send, delegation, withdraw reward, ibc tx. no wrong hash.

but live hub is incompatible with v6.0.2, @glnro

[mmulji-ic]

6.0.2 is redundant now because of the consensus bug and should not be used, use 6.0.3.