Skip to content

Navigation Menu

Explore
By company size
By use case
By industry
View all solutions
Topics
- AI
- DevOps
- Security
- Software Development
- View all
Explore
- GitHub Sponsors
  Fund open source developers
- The ReadME Project
  GitHub community articles
Repositories
- Enterprise platform
  AI-powered developer platform
Available add-ons
Pricing

Search code, repositories, users, issues, pull requests...

Search

Clear

Search syntax tips

Provide feedback

We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

Saved searches

Use saved searches to filter your results more quickly

Name

Query

To see all available qualifiers, see our documentation.

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.

Dismiss alert

filecoin-project / lotus Public

Notifications You must be signed in to change notification settings
Fork 1.3k
Star 2.9k

Code
Issues 593
Pull requests 63
Discussions
Actions
Projects 1
Security
Insights

Additional navigation options

Code
Issues
Pull requests
Discussions
Actions
Projects
Security
Insights

hot/cold blockstore segregation (aka. splitstore)#4992

Merged

magik6k merged 150 commits intonextfilecoin-project/lotus:nextfrom feat/splitstorefilecoin-project/lotus:feat/splitstoreCopy head branch name to clipboard

Mar 8, 2021

+2,499-20

Conversation Commits150 (150)Checks Files changed

Commits

Commits on Mar 5, 2021

splitstore struct and Blockstore interface implementation
vyzo
committed
compaction algorithm
vyzo
committed
trigger compaction from head changes
vyzo
committed
track base epoch in metadata ds
vyzo
committed
fix off by 1 in marking
vyzo
committed
use dual live set marking algorithm to keep all hotly reachable objects in the hotstore
vyzo
committed
satisfy linter
vyzo
committed
close keys channel when dome emitting keys
vyzo
committed
no need to import go-ipfs-blockstore, lib/blockstore will do
vyzo
committed
avoid race with compacting state variable
vyzo
committed
add some logging
vyzo
committed
move splitstore implementation to its own directory
vyzo
committed
stubs for tracking store and live set
vyzo
committed
gomod: get lmdb-go
vyzo
committed
implement lmdb-backed LiveSet
vyzo
committed
liveset unit test
vyzo
committed
quiet linter
vyzo
committed
implement LMDB-backed tracking store
vyzo
committed
add tracking store test
vyzo
committed
quiet linter
vyzo
committed
protect against potential data races
vyzo
committed
fallback to coldstore if snooping fails.
vyzo
committed
add TODO note about map size
vyzo
committed
simplify Has
vyzo
committed
use CAS for compacting state
vyzo
committed
go get go-bs-lmdb
vyzo
committed
splitstore constructor
vyzo
committed
hook splitstore into DI
vyzo
committed
fix lotus-shed build
vyzo
committed
add nil check for curTs -- some tests don't have chain state
vyzo
committed
appease linter
vyzo
committed
deal with MDB_KEY_EXIST errors
vyzo
committed
handle MDB_KEYEXIST in liveset marking
vyzo
committed
go get [email protected]
vyzo
committed
better handling of MDB_KEYEXIST in Put
vyzo
committed
make CompactionThreshold a var to fix lotus-soup build
vyzo
committed
fix max readers for tracking store
vyzo
committed
adjust walk boundaries for marking
vyzo
committed
temporary log level for splitstore to DEBUG
vyzo
committed
don't set max readers for livesets
vyzo
committed
update go-bs-lmdb and migrate to ledgerwatch/lmdb-go.

raulk
authored and
vyzo
committed
go get [email protected]
vyzo
committed
don't set MaxReaders for tracking store
vyzo
committed
retry on MDB_READERS_FULL errors
vyzo
committed
log MDB_READERS_FULL retries
vyzo
committed
fix potential panic with max readers retry and cursor channel
vyzo
committed
use xerrors instead of fmt.Errorf
vyzo
committed
return annotated xerrors where appropriate
vyzo
committed
fix snoop test
vyzo
committed
adjust hot store options
vyzo
committed
adjust hot store options, redux.
vyzo
committed
gomod:update go-bs-lmdb to v1.0.3
vyzo
committed
set lmdb max readers retry delay to 1ms
vyzo
committed
crank up blockstore max readers to 16K, reduce retry delays to 10us
vyzo
committed
flag to enable GC during compaction, disabled for now
vyzo
committed
make hot store DI injectable in the split store, default to badger.
vyzo
committed
get rid of goroutine iteration in tracking store; long live ForEach
vyzo
committed
amend confusing comment
vyzo
committed
use CompactionCold epochs for delinating the cold epoch cliff
vyzo
committed
walk snapshot the same way snapshot exporting does; skip old msgs and receipts by default.
vyzo
committed
handle consistency edge case
vyzo
committed
more informative names for the hotstore directories
vyzo
committed
refactor lmdb specific snoop/liveset code into their own files
vyzo
committed
rewrite sweep logic to avoid doing writes/deletes nested in a read txn
vyzo
committed
handle MDB_KEY_EXIST in tracking store Puts
vyzo
committed
add comment
vyzo
committed
abstract tracking store and live set construction
vyzo
committed
propagate useLMDB option to splitstore through DI
vyzo
committed
gomod: use bolt
vyzo
committed
implement bolt-backed liveset
vyzo
committed
add test for bolt liveset
vyzo
committed
separate LMDB options for hotstore and tracking stores
vyzo
committed
implement bolt backed tracking store
vyzo
committed
test for bolt backed tracking store
vyzo
committed
add configuration for splitstore and default to a simple compaction algorithm
vyzo
committed
set NoSync option for bolt livesets
vyzo
committed
add Sync to the tracking store
vyzo
committed
compactSimple should walk the cold epoch at depth 1
vyzo
committed
better logging plus moving some code around
vyzo
committed
more accurate setting of skip params
vyzo
committed
structured log for beginning of compaction
vyzo
committed
batch deletion for purging the tracking store
vyzo
committed
add (salted) bloom filter liveset
vyzo
committed
batch move objects from coldstore to hotstore
vyzo
committed
comment nomenclature
vyzo
committed
size bloom filter for 50M objects
vyzo
committed
fix tests
vyzo
committed
rehash salted keys in bloom filter
vyzo
committed
use named constants for bloom filter parameters
vyzo
committed
remove stale references to lmdb from splitstore implementation
vyzo
committed
remove references to splitstore from lotus-shed
vyzo
committed
quiet the stupid linter
vyzo
committed
make compaction parameters variable
vyzo
committed
dynamically size bloom filters
vyzo
committed
adjust min bloom filter size
vyzo
committed
fix tests
vyzo
committed
fix post-rebase compilation errors
vyzo
committed
go mod tidy
vyzo
committed
split off lmdb support to a different branch.

raulk
authored and
vyzo
committed
move splitstore into blockstore package.

raulk
authored and
vyzo
committed
warm up splitstore at first head change notification
vyzo
committed
snake current tipset from head change notification
vyzo
committed
also estimate liveset size during warm up
vyzo
committed
use crypto/rand for bloom salt
vyzo
committed
remove dependency from blockstore/splitstore => chain/store.

raulk
authored and
vyzo
committed
renames and polish.

raulk
authored and
vyzo
committed
batch writes during warm up
vyzo
committed
increase batch size to 16K
vyzo
committed
rename liveset => markset; rename snoop => tracking store; docs.

raulk
authored and
vyzo
committed
improve comment accuracy
vyzo
committed
use sha256 for bloom key rehashing
vyzo
committed
use ioutil.TempDir for test directories
vyzo
committed
log consistency for full compaction
vyzo
committed
deduplicate code
vyzo
committed
storage miner doesn't need a splitstore
vyzo
committed
move Blockstore config to FullNode, rename to Chainstore and add default for HotStoreType
vyzo
committed
add DeleteMany to Blockstore interface
vyzo
committed
fix the situation with WrapIDStore
vyzo
committed
satisfy linter, use Prefix for common path of non inline CIDs
vyzo
committed
fix typo

vyzo
and
raulk
committed
add comment

vyzo
and
raulk
committed
fix FromDatastore to not do double adapting
vyzo
committed
batch delete during the cold purge
vyzo
committed
add startup log
vyzo
committed
improve startup log
vyzo
committed
don't delete in one giant batch, use smaller chunks of batchSize
vyzo
committed
don't ID wrap the hotstore
vyzo
committed
do tracker purge in smaller batches
vyzo
committed
overestimate markSetSize a bit
vyzo
committed
fix log
vyzo
committed
properly close snoop at shutdown
vyzo
committed
walk at boundary epoch, 2 finalities from current epoch, to find live objects
vyzo
committed
implement DeleteMany in union blockstore
vyzo
committed
fix loop condition in batch deletion
vyzo
committed
deduplicate code for batch deletion
vyzo
committed
save markSetSize
vyzo
committed
use atomics to demarkate critical section and limit close delay
vyzo
committed
don't panic on compaction errors
vyzo
committed
remove DEBUG log spam
vyzo
committed
add some metrics
vyzo
committed
use the right condition for triggering the miss metric
vyzo
committed
add splitstore unit test
vyzo
committed
quiet stupid linter
vyzo
committed

Commits on Mar 8, 2021

garbage collect hotstore after compaction
vyzo
committed
also gc in compactFull, not just compactSimple
vyzo
committed
rename GC to CollectGarbage, ignore badger.ErrNoRewrite
vyzo
committed
deduplicate code
vyzo
committed
Merge pull request #5744 from filecoin-project/feat/splitstore-gc
vyzo
authored
tune badger gc to repeated gc the value log until there is no rewrite
vyzo
committed
Merge pull request #5745 from filecoin-project/feat/splitstore-gc-tuning
magik6k
authored

Footer

© 2024 GitHub, Inc.

Footer navigation

Terms
Privacy
Security
Status
Docs
Contact

You can’t perform that action at this time.