Refactor the wallet server/hub database use only leveldb (drop sqlite) and safely handle reorgs

[jackrobison]

fixes #3143, #3142

Handling reorgs in sqlite is very difficult and would require completely refactoring the table structure to properly unwind blocks and unwind the changes to urls. As a result, the sqlite wallet server will have inconsistent results that get worse as reorgs occur.

To fix this, this PR implements the claimtrie in a key/value database (currently leveldb) such that each operation can be unwound.

Refactoring:

• unify the different leveldb databases into one (utxo, hist, headers, transactions) so that caches are shared and we aren't doing operations between threads when doing lookups
• remove the separate History class used for address histories, integrate it in to the LevelDB database class.
• move all leveldb prefixes to DB_PREFIXES enum, previously these were split across several files and it was hard to see what prefixes were used where
• atomic block advance and rewind (apply all changes for a new or undone block as one atomic write, for data integrity)
• maintain in memory reverse mapping of txids to tx numbers (to save on i/o)

New interfaces:

• PrefixDB
• RevertableOp, used serialize and deserialize a reversible representation of an operation to put or delete to a key/value record.
• PrefixRow, used pack and unpack the keys and values for a data type in the k/v database. Each 'table' in the database has a PrefixRow class to pack and unpack the items in it.

[belikor]

Hi, I'm looking into the code of the SDK in a general way, and noticed some compilation issues with plyvel due to missing libleveldb-dev. See #3349.

I noticed that currently plyvel is compiled from source, as it's in setup.py. Currently it only seems to be used in lbry.wallet.server.storage.

I'm not sure how plyvel is used but wouldn't it make sense to simply use a prepackaged version of plyvel? In Ubuntu the package is in python3-plyvel, and in Arch it is in python-plyvel. I was thinking about mentioning the package as a dependency in INSTALL.md, and removing the variable PLYVEL from setup.py but I'm not sure if this affects anything in the code.

I tested by not installing plyvel at all, and then installing and running the unit tests; they seemed to work okay without referencing plyvel at all. So I'm wondering if the current code doesn't use plyvel, and it will only use it once this pull request is merged.

[belikor]

@jackrobison about my previous comment; I'm wondering if there is a particular reason to use plyvel 1.0.5.

The errors that I mentioned don't occur if we just set plyvel=1.2.0 or plyvel=1.3.0 in setup.py. This seems to be an easy solution. See #3357.

[jackrobison]

@belikor updated plyvel to 1.3.0

[belikor]

Nice.

Is this branch meant to be merged soon? If that's the case, I will close #3357. Otherwise I expect #3357 to be merged before, and you'll have to rebase this branch before merging.

[coveralls]

Coverage remained the same at 68.972% when pulling 43432a9 on leveldb-resolve into 84908ec on master.