Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hot/cold blockstore segregation (aka. splitstore) #4992

Merged
merged 150 commits into from
Mar 8, 2021
Merged
Changes from 1 commit
Commits
Show all changes
150 commits
Select commit Hold shift + click to select a range
6577cc8
splitstore struct and Blockstore interface implementation
vyzo Nov 24, 2020
c8f1139
compaction algorithm
vyzo Nov 24, 2020
b192adf
trigger compaction from head changes
vyzo Nov 24, 2020
fd08786
track base epoch in metadata ds
vyzo Nov 24, 2020
c2cc198
fix off by 1 in marking
vyzo Nov 24, 2020
2bed6c9
use dual live set marking algorithm to keep all hotly reachable objec…
vyzo Nov 24, 2020
b945747
satisfy linter
vyzo Nov 24, 2020
101e5c6
close keys channel when dome emitting keys
vyzo Nov 25, 2020
3083d80
no need to import go-ipfs-blockstore, lib/blockstore will do
vyzo Nov 25, 2020
c1b1a9c
avoid race with compacting state variable
vyzo Nov 25, 2020
2c9b58a
add some logging
vyzo Nov 25, 2020
17bc5fc
move splitstore implementation to its own directory
vyzo Nov 26, 2020
0bf1a78
stubs for tracking store and live set
vyzo Nov 26, 2020
df856b7
gomod: get lmdb-go
vyzo Nov 26, 2020
3f92a00
implement lmdb-backed LiveSet
vyzo Nov 26, 2020
5043f31
liveset unit test
vyzo Nov 26, 2020
83f8a0a
quiet linter
vyzo Nov 26, 2020
0d7476c
implement LMDB-backed tracking store
vyzo Nov 26, 2020
4763397
add tracking store test
vyzo Nov 26, 2020
da47883
quiet linter
vyzo Nov 26, 2020
d20cbc0
protect against potential data races
vyzo Nov 29, 2020
5db314f
fallback to coldstore if snooping fails.
vyzo Nov 29, 2020
37e391f
add TODO note about map size
vyzo Nov 29, 2020
0af7b16
simplify Has
vyzo Nov 29, 2020
b0f48b5
use CAS for compacting state
vyzo Nov 29, 2020
e87ce6c
go get go-bs-lmdb
vyzo Dec 1, 2020
e07c6c7
splitstore constructor
vyzo Dec 1, 2020
622b4f7
hook splitstore into DI
vyzo Dec 1, 2020
3912694
fix lotus-shed build
vyzo Dec 1, 2020
facdc55
add nil check for curTs -- some tests don't have chain state
vyzo Dec 1, 2020
f44cf0f
appease linter
vyzo Dec 1, 2020
843fd09
deal with MDB_KEY_EXIST errors
vyzo Dec 1, 2020
ce41e39
handle MDB_KEYEXIST in liveset marking
vyzo Dec 1, 2020
3f8da19
go get go-bs-lmdb@v0.0.3
vyzo Dec 1, 2020
6e51e6d
better handling of MDB_KEYEXIST in Put
vyzo Dec 1, 2020
1a23b1f
make CompactionThreshold a var to fix lotus-soup build
vyzo Dec 1, 2020
76d6edb
fix max readers for tracking store
vyzo Dec 1, 2020
8b00875
adjust walk boundaries for marking
vyzo Dec 1, 2020
58a8434
temporary log level for splitstore to DEBUG
vyzo Jan 13, 2021
5b4e6b7
don't set max readers for livesets
vyzo Jan 20, 2021
877ecab
update go-bs-lmdb and migrate to ledgerwatch/lmdb-go.
raulk Jan 25, 2021
5872f24
go get go-bs-lmdb@v1.0.2
vyzo Jan 26, 2021
2080e46
don't set MaxReaders for tracking store
vyzo Jan 29, 2021
c89ab1a
retry on MDB_READERS_FULL errors
vyzo Feb 1, 2021
b9f8a3d
log MDB_READERS_FULL retries
vyzo Feb 1, 2021
d91b60d
fix potential panic with max readers retry and cursor channel
vyzo Feb 1, 2021
ea05fd9
use xerrors instead of fmt.Errorf
vyzo Feb 10, 2021
cdf5bd0
return annotated xerrors where appropriate
vyzo Feb 10, 2021
69a88d4
fix snoop test
vyzo Feb 10, 2021
ca8a673
adjust hot store options
vyzo Feb 11, 2021
874ecd3
adjust hot store options, redux.
vyzo Feb 11, 2021
723e48b
gomod:update go-bs-lmdb to v1.0.3
vyzo Feb 11, 2021
95befa1
set lmdb max readers retry delay to 1ms
vyzo Feb 11, 2021
f6c930d
crank up blockstore max readers to 16K, reduce retry delays to 10us
vyzo Feb 13, 2021
7044e62
flag to enable GC during compaction, disabled for now
vyzo Feb 26, 2021
a586d42
make hot store DI injectable in the split store, default to badger.
vyzo Feb 26, 2021
842ec43
get rid of goroutine iteration in tracking store; long live ForEach
vyzo Feb 26, 2021
d44719d
amend confusing comment
vyzo Feb 26, 2021
5068d51
use CompactionCold epochs for delinating the cold epoch cliff
vyzo Feb 26, 2021
31268ba
walk snapshot the same way snapshot exporting does; skip old msgs and…
vyzo Feb 26, 2021
8e12377
handle consistency edge case
vyzo Feb 26, 2021
99c7d8e
more informative names for the hotstore directories
vyzo Feb 26, 2021
ee751f8
refactor lmdb specific snoop/liveset code into their own files
vyzo Feb 26, 2021
9977f5c
rewrite sweep logic to avoid doing writes/deletes nested in a read txn
vyzo Feb 26, 2021
e794451
handle MDB_KEY_EXIST in tracking store Puts
vyzo Feb 27, 2021
8f0ddac
add comment
vyzo Feb 27, 2021
923a3db
abstract tracking store and live set construction
vyzo Feb 27, 2021
68b6f91
propagate useLMDB option to splitstore through DI
vyzo Feb 27, 2021
cb1789e
gomod: use bolt
vyzo Feb 27, 2021
27a9b97
implement bolt-backed liveset
vyzo Feb 27, 2021
2c1a978
add test for bolt liveset
vyzo Feb 27, 2021
b839947
separate LMDB options for hotstore and tracking stores
vyzo Feb 27, 2021
f1c61c4
implement bolt backed tracking store
vyzo Feb 27, 2021
2e4d45e
test for bolt backed tracking store
vyzo Feb 27, 2021
73259aa
add configuration for splitstore and default to a simple compaction a…
vyzo Feb 27, 2021
364076c
set NoSync option for bolt livesets
vyzo Feb 27, 2021
783dcda
add Sync to the tracking store
vyzo Feb 27, 2021
2f26026
compactSimple should walk the cold epoch at depth 1
vyzo Feb 27, 2021
2426ffb
better logging plus moving some code around
vyzo Feb 27, 2021
e52c709
more accurate setting of skip params
vyzo Feb 27, 2021
09cd117
structured log for beginning of compaction
vyzo Feb 27, 2021
aba6530
batch deletion for purging the tracking store
vyzo Feb 27, 2021
97abbe1
add (salted) bloom filter liveset
vyzo Feb 28, 2021
4cc672d
batch move objects from coldstore to hotstore
vyzo Feb 28, 2021
f4c6bc6
comment nomenclature
vyzo Feb 28, 2021
f5ce795
size bloom filter for 50M objects
vyzo Feb 28, 2021
8884920
fix tests
vyzo Feb 28, 2021
44aadb9
rehash salted keys in bloom filter
vyzo Feb 28, 2021
f62999d
use named constants for bloom filter parameters
vyzo Feb 28, 2021
05fee27
remove stale references to lmdb from splitstore implementation
vyzo Feb 28, 2021
e582f0b
remove references to splitstore from lotus-shed
vyzo Feb 28, 2021
7587ab6
quiet the stupid linter
vyzo Feb 28, 2021
5639261
make compaction parameters variable
vyzo Feb 28, 2021
cae5ddc
dynamically size bloom filters
vyzo Feb 28, 2021
99c6e4f
adjust min bloom filter size
vyzo Feb 28, 2021
3282f85
fix tests
vyzo Feb 28, 2021
0fc2f3a
fix post-rebase compilation errors
vyzo Mar 1, 2021
3733456
go mod tidy
vyzo Mar 1, 2021
1b51c10
split off lmdb support to a different branch.
raulk Mar 1, 2021
1a804fb
move splitstore into blockstore package.
raulk Mar 1, 2021
cb36d5b
warm up splitstore at first head change notification
vyzo Mar 1, 2021
748dd96
snake current tipset from head change notification
vyzo Mar 1, 2021
e612fff
also estimate liveset size during warm up
vyzo Mar 1, 2021
b9400c5
use crypto/rand for bloom salt
vyzo Mar 1, 2021
b1b452b
remove dependency from blockstore/splitstore => chain/store.
raulk Mar 1, 2021
8cfba5b
renames and polish.
raulk Mar 1, 2021
ce68b9b
batch writes during warm up
vyzo Mar 1, 2021
48f2533
increase batch size to 16K
vyzo Mar 1, 2021
4b1e1f4
rename liveset => markset; rename snoop => tracking store; docs.
raulk Mar 2, 2021
f651f43
improve comment accuracy
vyzo Mar 2, 2021
35d466d
use sha256 for bloom key rehashing
vyzo Mar 2, 2021
68213a9
use ioutil.TempDir for test directories
vyzo Mar 2, 2021
5184bc5
log consistency for full compaction
vyzo Mar 2, 2021
c762536
deduplicate code
vyzo Mar 2, 2021
6014273
storage miner doesn't need a splitstore
vyzo Mar 2, 2021
dd0c308
move Blockstore config to FullNode, rename to Chainstore and add defa…
vyzo Mar 2, 2021
86b73d6
add DeleteMany to Blockstore interface
vyzo Mar 2, 2021
8a55b73
fix the situation with WrapIDStore
vyzo Mar 2, 2021
2ff5aec
satisfy linter, use Prefix for common path of non inline CIDs
vyzo Mar 2, 2021
86fdad2
fix typo
vyzo Mar 2, 2021
ab52e34
add comment
vyzo Mar 2, 2021
4c05ec2
fix FromDatastore to not do double adapting
vyzo Mar 2, 2021
06d8ea1
batch delete during the cold purge
vyzo Mar 2, 2021
006c55a
add startup log
vyzo Mar 2, 2021
70ebb2a
improve startup log
vyzo Mar 2, 2021
d2d0980
don't delete in one giant batch, use smaller chunks of batchSize
vyzo Mar 2, 2021
6b8c60a
don't ID wrap the hotstore
vyzo Mar 2, 2021
6b680d1
do tracker purge in smaller batches
vyzo Mar 2, 2021
11b2f41
overestimate markSetSize a bit
vyzo Mar 3, 2021
47d8c87
fix log
vyzo Mar 3, 2021
508fcb9
properly close snoop at shutdown
vyzo Mar 3, 2021
fdd8775
walk at boundary epoch, 2 finalities from current epoch, to find live…
vyzo Mar 3, 2021
98a7b88
implement DeleteMany in union blockstore
vyzo Mar 3, 2021
5fb6a90
fix loop condition in batch deletion
vyzo Mar 3, 2021
aff0f1e
deduplicate code for batch deletion
vyzo Mar 3, 2021
17be7d3
save markSetSize
vyzo Mar 5, 2021
9bd009d
use atomics to demarkate critical section and limit close delay
vyzo Mar 5, 2021
c58df3f
don't panic on compaction errors
vyzo Mar 5, 2021
99d2157
remove DEBUG log spam
vyzo Mar 5, 2021
2b32c2e
add some metrics
vyzo Mar 5, 2021
0a2f2cf
use the right condition for triggering the miss metric
vyzo Mar 5, 2021
09f5ba1
add splitstore unit test
vyzo Mar 5, 2021
e85391b
quiet stupid linter
vyzo Mar 5, 2021
8562a9b
garbage collect hotstore after compaction
vyzo Mar 8, 2021
52de95d
also gc in compactFull, not just compactSimple
vyzo Mar 8, 2021
3d1b855
rename GC to CollectGarbage, ignore badger.ErrNoRewrite
vyzo Mar 8, 2021
3bd7770
deduplicate code
vyzo Mar 8, 2021
c52dae4
Merge pull request #5744 from filecoin-project/feat/splitstore-gc
vyzo Mar 8, 2021
90741da
tune badger gc to repeated gc the value log until there is no rewrite
vyzo Mar 8, 2021
51ed4c7
Merge pull request #5745 from filecoin-project/feat/splitstore-gc-tuning
magik6k Mar 8, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add splitstore unit test
  • Loading branch information
vyzo committed Mar 5, 2021
commit 09f5ba177a936636bd94013d4cfef249829f2056
2 changes: 1 addition & 1 deletion blockstore/splitstore/splitstore.go
Original file line number Diff line number Diff line change
@@ -79,7 +79,7 @@ const (
type Config struct {
// TrackingStore is the type of tracking store to use.
//
// Supported values are: "bolt" (default if omitted).
// Supported values are: "bolt" (default if omitted), "mem" (for tests and readonly access).
TrackingStoreType string

// MarkSetType is the type of mark set to use.
248 changes: 248 additions & 0 deletions blockstore/splitstore/splitstore_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,248 @@
package splitstore

import (
"context"
"fmt"
"sync"
"testing"
"time"

"github.com/filecoin-project/go-state-types/abi"
"github.com/filecoin-project/lotus/blockstore"
"github.com/filecoin-project/lotus/chain/types"
"github.com/filecoin-project/lotus/chain/types/mock"

cid "github.com/ipfs/go-cid"
datastore "github.com/ipfs/go-datastore"
logging "github.com/ipfs/go-log/v2"
)

func init() {
CompactionThreshold = 5
CompactionCold = 1
CompactionBoundary = 2
logging.SetLogLevel("splitstore", "DEBUG")
}

func testSplitStore(t *testing.T, cfg *Config) {
t.Helper()

chain := &mockChain{}
// genesis
genBlock := mock.MkBlock(nil, 0, 0)
genTs := mock.TipSet(genBlock)
chain.push(genTs)

// the myriads of stores
ds := datastore.NewMapDatastore()
hot := blockstore.NewMemorySync()
cold := blockstore.NewMemorySync()

// put the genesis block to cold store
blk, err := genBlock.ToStorageBlock()
if err != nil {
t.Fatal(err)
}

err = cold.Put(blk)
if err != nil {
t.Fatal(err)
}

// open the splitstore
ss, err := Open("", ds, hot, cold, cfg)
if err != nil {
t.Fatal(err)
}
defer ss.Close()

err = ss.Start(chain)
if err != nil {
t.Fatal(err)
}

// make some tipsets, but not enough to cause compaction
mkBlock := func(curTs *types.TipSet, i int) *types.TipSet {
blk := mock.MkBlock(curTs, uint64(i), uint64(i))
sblk, err := blk.ToStorageBlock()
if err != nil {
t.Fatal(err)
}
err = ss.Put(sblk)
if err != nil {
t.Fatal(err)
}
ts := mock.TipSet(blk)
chain.push(ts)

return ts
}

mkGarbageBlock := func(curTs *types.TipSet, i int) {
blk := mock.MkBlock(curTs, uint64(i), uint64(i))
sblk, err := blk.ToStorageBlock()
if err != nil {
t.Fatal(err)
}
err = ss.Put(sblk)
if err != nil {
t.Fatal(err)
}
}

curTs := genTs
for i := 1; i < 5; i++ {
curTs = mkBlock(curTs, i)
}

mkGarbageBlock(genTs, 1)

// count objects in the cold and hot stores
ctx, cancel := context.WithCancel(context.Background())
defer cancel()

countBlocks := func(bs blockstore.Blockstore) int {
count := 0
ch, err := bs.AllKeysChan(ctx)
if err != nil {
t.Fatal(err)
}
for _ = range ch {
count++
}
return count
}

coldCnt := countBlocks(cold)
hotCnt := countBlocks(hot)

if coldCnt != 1 {
t.Fatalf("expected %d blocks, but got %d", 1, coldCnt)
}

if hotCnt != 4 {
t.Fatalf("expected %d blocks, but got %d", 4, hotCnt)
}

// trigger a compaction
for i := 5; i < 10; i++ {
curTs = mkBlock(curTs, i)
time.Sleep(time.Second)
}

coldCnt = countBlocks(cold)
hotCnt = countBlocks(hot)

if !cfg.EnableFullCompaction {
if coldCnt != 5 {
t.Fatalf("expected %d cold blocks, but got %d", 5, coldCnt)
}

if hotCnt != 5 {
t.Fatalf("expected %d hot blocks, but got %d", 5, hotCnt)
}
}

if cfg.EnableFullCompaction && !cfg.EnableGC {
if coldCnt != 3 {
t.Fatalf("expected %d cold blocks, but got %d", 3, coldCnt)
}

if hotCnt != 7 {
t.Fatalf("expected %d hot blocks, but got %d", 7, hotCnt)
}
}

if cfg.EnableFullCompaction && cfg.EnableGC {
if coldCnt != 2 {
t.Fatalf("expected %d cold blocks, but got %d", 2, coldCnt)
}

if hotCnt != 7 {
t.Fatalf("expected %d hot blocks, but got %d", 7, hotCnt)
}
}
}

func TestSplitStoreSimpleCompaction(t *testing.T) {
testSplitStore(t, &Config{TrackingStoreType: "mem"})
}

func TestSplitStoreFullCompactionWithoutGC(t *testing.T) {
testSplitStore(t, &Config{
TrackingStoreType: "mem",
EnableFullCompaction: true,
})
}

func TestSplitStoreFullCompactionWithGC(t *testing.T) {
testSplitStore(t, &Config{
TrackingStoreType: "mem",
EnableFullCompaction: true,
EnableGC: true,
})
}

type mockChain struct {
sync.Mutex
tipsets []*types.TipSet
listener func(revert []*types.TipSet, apply []*types.TipSet) error
}

func (c *mockChain) push(ts *types.TipSet) {
c.Lock()
c.tipsets = append(c.tipsets, ts)
c.Unlock()

if c.listener != nil {
err := c.listener(nil, []*types.TipSet{ts})
if err != nil {
log.Errorf("mockchain: error dispatching listener: %s", err)
}
}
}

func (c *mockChain) GetTipsetByHeight(_ context.Context, epoch abi.ChainEpoch, _ *types.TipSet, _ bool) (*types.TipSet, error) {
c.Lock()
defer c.Unlock()

iEpoch := int(epoch)
if iEpoch > len(c.tipsets) {
return nil, fmt.Errorf("bad epoch %d", epoch)
}

return c.tipsets[iEpoch-1], nil
}

func (c *mockChain) GetHeaviestTipSet() *types.TipSet {
c.Lock()
defer c.Unlock()

return c.tipsets[len(c.tipsets)-1]
}

func (c *mockChain) SubscribeHeadChanges(change func(revert []*types.TipSet, apply []*types.TipSet) error) {
c.listener = change
}

func (c *mockChain) WalkSnapshot(_ context.Context, ts *types.TipSet, epochs abi.ChainEpoch, _ bool, _ bool, f func(cid.Cid) error) error {
c.Lock()
defer c.Unlock()

start := int(ts.Height()) - 1
end := start - int(epochs)
if end < 0 {
end = -1
}
for i := start; i > end; i-- {
ts := c.tipsets[i]
for _, cid := range ts.Cids() {
err := f(cid)
if err != nil {
return err
}
}
}

return nil
}
75 changes: 75 additions & 0 deletions blockstore/splitstore/tracking.go
Original file line number Diff line number Diff line change
@@ -2,6 +2,7 @@ package splitstore

import (
"path/filepath"
"sync"

"golang.org/x/xerrors"

@@ -28,7 +29,81 @@ func OpenTrackingStore(path string, ttype string) (TrackingStore, error) {
switch ttype {
case "", "bolt":
return OpenBoltTrackingStore(filepath.Join(path, "tracker.bolt"))
case "mem":
return NewMemTrackingStore(), nil
default:
return nil, xerrors.Errorf("unknown tracking store type %s", ttype)
}
}

// NewMemTrackingStore creates an in-memory tracking store.
// This is only useful for test or situations where you don't want to open the
// real tracking store (eg concurrent read only access on a node's datastore)
func NewMemTrackingStore() *MemTrackingStore {
return &MemTrackingStore{tab: make(map[cid.Cid]abi.ChainEpoch)}
}

// MemTrackingStore is a simple in-memory tracking store
type MemTrackingStore struct {
sync.Mutex
tab map[cid.Cid]abi.ChainEpoch
}

var _ TrackingStore = (*MemTrackingStore)(nil)

func (s *MemTrackingStore) Put(cid cid.Cid, epoch abi.ChainEpoch) error {
s.Lock()
defer s.Unlock()
s.tab[cid] = epoch
return nil
}

func (s *MemTrackingStore) PutBatch(cids []cid.Cid, epoch abi.ChainEpoch) error {
s.Lock()
defer s.Unlock()
for _, cid := range cids {
s.tab[cid] = epoch
}
return nil
}

func (s *MemTrackingStore) Get(cid cid.Cid) (abi.ChainEpoch, error) {
s.Lock()
defer s.Unlock()
epoch, ok := s.tab[cid]
if ok {
return epoch, nil
}
return 0, xerrors.Errorf("missing tracking epoch for %s", cid)
}

func (s *MemTrackingStore) Delete(cid cid.Cid) error {
s.Lock()
defer s.Unlock()
delete(s.tab, cid)
return nil
}

func (s *MemTrackingStore) DeleteBatch(cids []cid.Cid) error {
s.Lock()
defer s.Unlock()
for _, cid := range cids {
delete(s.tab, cid)
}
return nil
}

func (s *MemTrackingStore) ForEach(f func(cid.Cid, abi.ChainEpoch) error) error {
s.Lock()
defer s.Unlock()
for cid, epoch := range s.tab {
err := f(cid, epoch)
if err != nil {
return err
}
}
return nil
}

func (s *MemTrackingStore) Sync() error { return nil }
func (s *MemTrackingStore) Close() error { return nil }