Localstore integration #1282

janos · 2019-03-05T14:51:48Z

This PR replaces LDBStore with new localstore DB in all parts of the code.

Work needed to be done is migration from legacy data store #1297 and logging and tracing in localstore package. Beside this missing parts, this PR should provide a functional swarm node starting from a blank database.

janos · 2019-03-05T15:00:31Z

PR ethereum/go-ethereum#19215 was opened by mistake on ethereum, which can be reused if this PR is accepted.

zelig

would it not make sense to split this PR and extract the Has/shed related changes?

swarm/storage/localstore/mode_has.go

swarm/storage/localstore/localstore.go

janos · 2019-03-06T06:30:10Z

would it not make sense to split this PR and extract the Has/shed related changes?

Yes, I will create a new PR, Has is pretty isolated change.

zelig · 2019-03-14T06:10:41Z

cmd/swarm/db.go

@@ -135,13 +135,10 @@ func dbImport(ctx *cli.Context) {
 	log.Info(fmt.Sprintf("successfully imported %d chunks", count))
 }

-func openLDBStore(path string, basekey []byte) (*storage.LDBStore, error) {
+func openLDBStore(path string, basekey []byte) (*localstore.DB, error) {


could this not be in the localstore pkg as one of the constructors?

Actually, I am not sure that we need it. It only checks if goleveldb data is present in a directory. Is it really necessary to have that check for export and import?

swarm/network/stream/common_test.go

swarm/storage/localstore/subscription_pull.go

janos · 2019-03-26T14:45:16Z

I am blocking this PR as it presents much more problems then improvements.

From gitter https://gitter.im/ethersphere/hq?at=5c9a376081b15c5e4b8ca43a:

Elad @justelad 15:29
right now i have made so little progress since localstore is, at the moment, some freak mutant hybrid of the old localstore and the new localstore

Please, do not review this PR until an alternative solution is presented.

zelig · 2019-03-27T18:13:48Z

swarm/network/stream/delivery.go

+		if peerPO > po {
+			mode = chunk.ModePutRequest
+		} else {
+			mode = chunk.ModePutSync


@justelad

no its not generating more redundancy the rest i dont understand

it is easy to construct test cases. this is about correctness

zelig · 2019-03-27T18:20:03Z

swarm/network/stream/delivery.go

-		} else {
+		depth := d.kad.NeighbourhoodDepth()
+		// is the chunk within our area of responsibility?
+		if po < depth {


if po >= depth || peerPO < po { mode = chunk.ModePutSync } else { mode = chunk.ModePutRequest } ```

Yes, this is more compact representation. I thought that the current one was easier to reason about, but this is good with a comment, as well.

that is good but you got the comparison wrong, note
po >= depth -> sync

Thanks Viktor.

janos · 2019-03-29T08:49:51Z

Thanks for reviews. @holisticode I have set this PR ready for review, again. Before this gets merged, maybe we should decide about the migration PR that @justelad is working on. If this PR is merged, and released without the migration part, users will not have an option to keep their existing data.

janos · 2019-03-29T08:52:24Z

swarm/storage/localstore/localstore.go

 	// create a pull syncing triggers used by SubscribePull function
 	db.pullTriggers = make(map[uint8][]chan struct{})
 	// push index contains as yet unsynced chunks
-	db.pushIndex, err = db.shed.NewIndex("StoredTimestamp|Hash->nil", shed.IndexFuncs{
+	db.pushIndex, err = db.shed.NewIndex("StoreTimestamp|Hash->Tags", shed.IndexFuncs{


@zelig @justelad do we need to revert this index not to have Tags and remove tags from shed Item?

@janos we definitely need a way to persist tags and I think that this is the correct place to do so. If anything, tags should be removed from chunk.Chunk

chunk.Chunk does not have reference to Tags, currently. Are you referring to shed.Item or some other code?

…the same chunk

janos · 2019-03-29T13:31:31Z

@zelig @justelad I have added a check if chunk is already stored 2e24e99

holisticode · 2019-03-29T19:14:59Z

swarm/network/stream/common_test.go

@@ -141,13 +141,13 @@ func netStoreAndDeliveryWithAddr(ctx *adapters.ServiceContext, bucket *sync.Map,

 	cleanup := func() {
 		netStore.Close()
-		os.RemoveAll(datadir)
+		localStoreCleanup()


why not localStore.Close() too?

netStore.Close() is calling localstore.Close()

holisticode · 2019-03-29T19:15:46Z

swarm/network/stream/common_test.go

 	teardown := func() {
 		streamer.Close()
+		intervalsStore.Close()
+		netStore.Close()
 		removeDataDir()


localStore.Close()?

netStore.Close() is calling localstore.Close()

This reverts commit d8c648b.

This reverts commit b828a34.

…ean on chunk exists (#1330) * swarm/chunk, swarm/storage: change chunk store interface to return bool on item exists * swarm/storage: adjust put test to cover `exists` param * swarm/storage/localstore: fix test assertions

nonsense · 2019-04-11T13:38:30Z

@janos I think we can close this. I think we should consider this merged, since it is in the swarm-rather-stable branch, and we just need to decide on specific timing regarding releasing it.

janos · 2019-04-11T13:48:40Z

Yes, thanks, Anton. Closing this PR since it is merged into https://github.com/ethersphere/go-ethereum/tree/swarm-rather-stable branch.

janos added 9 commits February 27, 2019 11:04

swarm/storage/localstore: add hasser

df7d887

swarm: merge branch 'master' into localstore-storage-integration

c0a4b1b

swarm: merge branch 'master' into localstore-storage-integration

8381e38

cmd/swarm, swarm: integrate localstore partially

5ac0f79

swarm: merge branch 'master' into localstore-storage-integration

a96aeeb

swarm/network/stream: start of localstore integration

44633ce

swarm/storage/localstore: add TestExportImport and fix Export function

3e6e03a

cmd/swarm: re-enable TestCLISwarmExportImport

6ce3481

swarm: extend chunk.Store

5e69e3d

janos added the in progress label Mar 5, 2019

janos self-assigned this Mar 5, 2019

janos added 2 commits March 5, 2019 16:24

swarm/storage/feed: fix TestValidatorInStore

5d7f169

swarm: rename ModeGetFeedLookup to ModeGetLookup

f0a8d3a

zelig reviewed Mar 5, 2019

View reviewed changes

swarm/storage/localstore/mode_has.go Outdated Show resolved Hide resolved

swarm/storage/localstore/localstore.go Show resolved Hide resolved

janos added 8 commits March 6, 2019 10:12

swarm: minor updates to chunk, storage and localstore packages

5474d30

swarm: merge branch 'master' into localstore-storage-integration

9ca71f4

swarm/shed: add vector uint64 field

83b64ce

cmd/swarm: fix merge issue

547f30b

swarm: use BinID for pull syncing index

970b305

swarm/network, swarm/storage/localstore: SetNextBatch pull subscription

5a669c6

swarm/network/stream: fix TestSyncerSimulation

4594130

swarm: change localstore SubscribePull function signature

4af1d7a

zelig added storage-refactor labels Mar 14, 2019

zelig reviewed Mar 14, 2019

View reviewed changes

swarm/storage/localstore: fix SubscribePull comment

4df51ee

swarm: merge branch 'master' into localstore-storage-integration

c9d679d

janos requested review from acud March 26, 2019 12:52

swarm/storage/localstore: implement backward compatible Import function

2fe747e

janos added discussion and removed ready for review labels Mar 26, 2019

janos and others added 2 commits March 27, 2019 09:22

swamr/network/stream: fix handleChunkDeliveryMsg

3e82234

swarm/storage/localstore: remove getters, setters and putters

4c14a09

zelig suggested changes Mar 27, 2019

View reviewed changes

swarm/network/stream: update handleChunkDeliveryMsg

31f4b0d

zelig approved these changes Mar 28, 2019

View reviewed changes

acud approved these changes Mar 28, 2019

View reviewed changes

janos added ready for review and removed discussion labels Mar 29, 2019

janos commented Mar 29, 2019

View reviewed changes

swarm/storage/localstore: only one item in pull and push indexes for …

2e24e99

…the same chunk

holisticode reviewed Mar 29, 2019

View reviewed changes

acud added 4 commits April 8, 2019 20:53

Revert "swarm/storage/localstore: TestDB_pushIndex_Tags"

9c79e26

This reverts commit d8c648b.

Revert "swarm/storage, swarm/shed: add support for persisting push tags"

ca7dcc2

This reverts commit b828a34.

cmd/swarm: new localstore database migration (#1297)

904df7a

janos closed this Apr 11, 2019

acud deleted the localstore-storage-integration branch May 5, 2019 18:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Localstore integration #1282

Localstore integration #1282

janos commented Mar 5, 2019 •

edited

Loading

janos commented Mar 5, 2019

zelig left a comment

janos commented Mar 6, 2019

zelig Mar 14, 2019

janos Mar 14, 2019

janos commented Mar 26, 2019

zelig Mar 27, 2019

zelig Mar 27, 2019

janos Mar 27, 2019

zelig Mar 28, 2019

janos Mar 28, 2019

janos commented Mar 29, 2019

janos Mar 29, 2019

acud Apr 1, 2019

janos Apr 2, 2019

janos commented Mar 29, 2019

holisticode Mar 29, 2019

janos Mar 29, 2019

holisticode Mar 29, 2019

janos Mar 29, 2019

nonsense commented Apr 11, 2019

janos commented Apr 11, 2019

Localstore integration #1282

Localstore integration #1282

Conversation

janos commented Mar 5, 2019 • edited Loading

janos commented Mar 5, 2019

zelig left a comment

Choose a reason for hiding this comment

janos commented Mar 6, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

janos commented Mar 26, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

janos commented Mar 29, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

janos commented Mar 29, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nonsense commented Apr 11, 2019

janos commented Apr 11, 2019

janos commented Mar 5, 2019 •

edited

Loading