hot/cold blockstore segregation (aka. splitstore)

[vyzo]

Fixes #4753.

This is the implementation of splitstore, a hot/cold segregated blockstore that can be enabled with a feature flag.

On top of #5695

How to enable:
In .lotus/config.toml add the following section:

[Chainstore]
  EnableSplitstore = true

TODO:

• dynamic bloom filter sizing
• move lmdb support out to a branch, to reduce surface and keep it out of the dependency graph.
  • moved to [EXPERIMENTAL] lmdb support for splitstore #5699
• appropriate options for badger hotstore; it currently uses as the coldstore, but we need to check if those actually make sense and tune; especially verify that value log is GC'ed and what is the correct setting for the GC threshold.
• batch delete support in blockstores.
• splitstore unit test.
• Properly open the Splitstore (not just the Coldstore) in lotus-shed when enabled by the configuration.

Follow up:

• optimize WalkSnapshot -- see Chain export inefficiencies #4994
• implement BatchDelete in Blockstores -- done in add DeleteMany to Blockstore interface #5703

[raulk]

Initial sweep; need to delve into the migration logic proper.

[ribasushi]

@vyzo @raulk from a conversation elsewhere I have the following requirement for this:

I'd want to undo the split in "capture" mode - there must be a way for both cold and hot must write to the same place without an itermediary/delay
this is pretty critical
this doesn't necesarily need to be an exposed config option: for the time being the capture-store needs a few codechanges in lotus anyway, so being ale to replace couple lines in fsrepo is enough for me
the ask is specifically for the interfaces not to diverge too far from the basis ipfs-blockstore one

[vyzo]

we can simply add a config option that disables gc -- you will still use the splitstore, but persist all objects without deleting unreachable ones.

[vyzo]

RIPed the freecache stuff and rebased directly to master; will fix conflicts.

[ribasushi]

we can simply add a config option that disables gc -- you will still use the splitstore, but persist all objects without deleting unreachable ones.

For the specific case I need this for this is not relevant: both underlying blockstores need to be exchangeable with a single 3rd party one, in order to allow for synchronous-ish storage. This is why I focus on "keep the interface close enough", as "no code changes" and "use 2 stores" is not an option in my case either way.

[vyzo]

Not sure I follow -- what do you want to exchange? You can simply use the coldstore.

[vyzo]

Also the blockstore interface is not changing.

[vyzo]

rebased on #5695

[vyzo]

rebased on next and fixed conflict.

[magik6k]

(test lgtm too)

[arajasek]

Broadly makes sense, and running well on my node for a few days