Auto-restart when network reset happens

[bowenwang1996]

Story

As a user who uses betanet and devnet, I would like to have auto-restart for devnet and betanet explorer when a network reset happens. It suffices to detect whether genesis hash changed periodically.

Acceptance Criteria

• Devnet explorer auto-restarts when devnet resets
• Betanet explorer auto-restarts when betanet resets.

[frol]

Implementation notes:

• Setup a 10-second timer on the backend checking for genesis_height match via EXPERIMENTAL_genesis_config API using sendJsonRpc on near-api-js. Store the previous value in memory and make sure that failing responses do not trigger reset.

• Let's pull the trigger if the last 6 times the genesis_time is different from the current genesis_time the block hash with the index of genesis_height is different from the current genesis block hash.

  NOTE: Let's assume that the first time on Explorer backend boot we query the current genesis block hash (this is not exactly correct, but let's do this for now, I expect we resolve this with Include information from genesis records #185 or Top level economic stats #189)

• Once we decided to reset, we should drop rename the database (remove rename the relevant file to include the current date & time and restart the process -- kill itself; we need to set up a supervisor to boot the backend instead of the naive bash script there)

[bowenwang1996]

@frol I believe that genesis time may not be properly updated right now. It is safer to always require the genesis block and obtain its hash to see whether that has changed.

[frol]

@bowenwang1996 That is genesis_height, right?

[bowenwang1996]

Yes

[frol]

Updated my post above.

[bowenwang1996]

@frol why do you need to check it 6 times before restarting?

[frol]

@bowenwang1996 resync is quite an expensive task, so I would prefer to give a minute before dropping the previously synced state.

[icerove]

@bowenwang1996 @frol I can not find genesis_height in the EXPERIMENTAL_genesis_config API. only see genesis_time

[frol]

@icerove It was introduced "recently" (near/nearcore#2248), so it did not make to the example output. It is there.

[ilblackdragon]

Let's not drop old data at all. We should keep it in the explorer.
Just start indexing when new genesis block have happened - now that we keep the index after reset - there should not be any problems with that.

[frol]

Well, Explorer is not self-sufficient, it relies on the data to be available from the node. Also, I think it is going to be confusing if you can see a blocks and transactions that don't exist on the current network. We can archive the data (instead of removing the DB, let's rename the file, and we don't actually need to wait a minute before doing so, as I proposed above to avoid dropping the data too fast, since having the backup we can revert the state if needed [updated the implementation notes above]).

[ilblackdragon]

It's not confusing if you think that it's the same network, just was re-snappshotted.
E.g. people expect old links to transactions to work, those transactions before reset to show up in the wallet, etc.

we also want to have old stuff for better metrics, from total tx over time, total blocks over time, total new accounts over time, etc. CC @potatodepaulo @vgrichina

[frol]

@ilblackdragon why people would not expect to do RPC call to fetch a block/transaction and receive the same output as they see on Explorer?

[frol]

Resolved in #313