Rough draft of SectorSet #333

ZenGround0 · 2019-06-07T16:36:23Z

Plowing ahead with these assumptions I've drafted a rough draft of the SectorSet spec. Please take a look.

The prose is not great or final-spec material but I'm hoping it gets across the basics of what I'm planning to implement based on #116 so that we can sync state and catch anything I'm missing or address inefficiencies.

ZenGround0 · 2019-06-07T16:37:13Z

data-structures.md

@@ -221,7 +221,107 @@ Bitfields are a set of bits encoded using a custom run length encoding: rle+.  r

 #### `SectorSet`

-TODO
+The `SectorSet` is an integer set implemented with a simple array mapped tree.
+Integer indexes range from 0 to infinity (TODO practical bounds / bounds implied by encoding?).


Is 64 bit int limitation OK?

Probably worth setting some maximum based on the maximum number of sectors we expect to impose on miners

ZenGround0 · 2019-06-07T16:37:47Z

data-structures.md

+the `Pointer`. There is also a `RootNode`.   The `Node` is as follows:
+```go
+type Node struct {
+     Pointers [S]*Pointer


Creating a new node always allocates S empty Pointers

ZenGround0 · 2019-06-07T16:38:22Z

data-structures.md

+```
+The `Pointer` is serialized as a cbor object (major type 5). The  `Value`
+is serialized { in some way } with `v` as its object key.  The `SectorMeta`
+type carries all needed sector metadata (TODO, define a type for commD commR


Is this (commR, commD)?

SectorMeta will need:

CommR so that we can generate PoSts (the generate_post routines takes as input all CommRs in the miner's proving set).

CommD for piece inclusion proof verification.

ZenGround0 · 2019-06-07T16:41:21Z

data-structures.md

+of nodes until reaching the leaf node and set the node's pointer Value at
+`index % childRange` to the insert value.
+
+##### Delete Value


Lack of bitfields and constant sized arrays might make this prohibitively expensive as we have search the whole array for emptyness (we could include a simple count to make this better). Also note that as specced out here users can arrive at two identical sets with different representations. Id appeciate comments on the importance of addressing these issues.

iterating over some small constant number of items shouldnt be prohibitively expensive. Since we have to parse bytes into a datastructure anyways, we can keep track of the item count in memory (but not for the serialized version)

ZenGround0 · 2019-06-07T16:41:53Z

data-structures.md

+`index % childRange` to the insert value.
+
+##### Delete Value
+


I'm not covering range deletes here. Let me know if that's a requirement for v1.

ZenGround0 · 2019-06-07T16:44:22Z

data-structures.md

+the root that the `SectorSet` has capacity for its existing indices and `b`.
+Pointers are then updated in these new nodes so that there is a path from
+the new node with the highest height to the existing node pointed to by root.
+Finally the root node updates to point to the node with highest height.


Note that as the SectorSet grows the first insert into the new region of the tree will start to require more and more overhead. Making this scale better probably requires a redesign of the datastructure (we could do something closer to the HAMT with more complexity to improve this). Is the current design acceptable or should we try to address this now?

logarithmic scaling is about optimal here.

whyrusleeping · 2019-06-10T22:54:01Z

data-structures.md

+     NodePointer Cid
+}
+```
+The `RootNode` is serialized as a cbor object (major type 5) where `NodePointer`


I dont' think the rootnode needs to be serialized ever.

whyrusleeping · 2019-06-10T22:54:29Z

data-structures.md

+is serialized { in some way } with `v` as its object key.  The `SectorMeta`
+type carries all needed sector metadata (TODO, define a type for commD commR
+pair and serialization).  A `Pointer` is conceptually a union type, it can 
+carry a `Link` or a `Value` but never both in a well formed `SectorSet`


whats the object key used for the Link?

whyrusleeping · 2019-06-10T22:59:41Z

data-structures.md

+Pseudocode:
+```go
+func (rn *RootNode) Lookup(index int) (v Anything, error) {
+     return recLookup(rn.MaxDepth, index)


why doesnt this mention the expansion?

oh, this is just for lookup, i see

dignifiedquire · 2019-07-02T10:41:30Z

data-structures.md

@@ -221,7 +221,107 @@ Bitfields are a set of bits encoded using a custom run length encoding: rle+.  r

 #### `SectorSet`

-TODO
+The `SectorSet` is an integer set implemented with a simple array mapped tree.


lets call the data structure AMT and say that a

type SectorSet {UInt:SectorMeta}<AMT>

dignifiedquire · 2019-07-03T08:53:29Z

Moved content to ipld/specs#137 and the definition of the type into #367

mishmosh · 2020-01-06T17:48:06Z

The big overhaul in Sept/Oct means none of these changed files exist anymore. But, [at least some of] this prose is still valuable in these new locations:

Rendering: https://filecoin-project.github.io/specs/#systems__filecoin_mining__sector__sectorset
Code: https://github.com/filecoin-project/specs/tree/master/src/systems/filecoin_mining/sector

Proposal: let's create a new PR with the prose that's still true & useful from this, and close this one. @ZenGround0 do you prefer to do it yourself? @sternhenri/I can also create a first pass for you to review.

Also, #367 contains followup work.

ZenGround0 · 2020-07-06T16:57:28Z

I think this became AMT regardless this is very out of date

Rough draft of SectorSet

df10316

ZenGround0 commented Jun 7, 2019

View reviewed changes

ZenGround0 requested review from whyrusleeping and dignifiedquire June 7, 2019 16:46

whyrusleeping reviewed Jun 10, 2019

View reviewed changes

ZenGround0 mentioned this pull request Jun 27, 2019

SectorSet: AMT design spike filecoin-project/venus#2887

Closed

pooja mentioned this pull request Jul 1, 2019

Complete spec for sector sets and related datastructures #298

Open

5 tasks

dignifiedquire reviewed Jul 2, 2019

View reviewed changes

dignifiedquire mentioned this pull request Jul 2, 2019

feat: define SectorSet #367

Closed

pooja mentioned this pull request Jul 30, 2019

TODO for August 6 #418

Closed

18 tasks

pooja mentioned this pull request Aug 27, 2019

TODO for September 3 #460

Closed

22 tasks

ZenGround0 closed this Jul 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rough draft of SectorSet #333

Rough draft of SectorSet #333

ZenGround0 commented Jun 7, 2019 •

edited by pooja

Loading

ZenGround0 Jun 7, 2019

whyrusleeping Jun 10, 2019

ZenGround0 Jun 7, 2019

ZenGround0 Jun 7, 2019

whyrusleeping Jun 10, 2019

laser Jun 10, 2019

ZenGround0 Jun 7, 2019

whyrusleeping Jun 10, 2019

ZenGround0 Jun 7, 2019

ZenGround0 Jun 7, 2019 •

edited

Loading

whyrusleeping Jun 10, 2019

whyrusleeping Jun 10, 2019

whyrusleeping Jun 10, 2019

whyrusleeping Jun 10, 2019

whyrusleeping Jun 10, 2019

dignifiedquire Jul 2, 2019

dignifiedquire commented Jul 3, 2019

mishmosh commented Jan 6, 2020

ZenGround0 commented Jul 6, 2020

Rough draft of SectorSet #333

Rough draft of SectorSet #333

Conversation

ZenGround0 commented Jun 7, 2019 • edited by pooja Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ZenGround0 Jun 7, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dignifiedquire commented Jul 3, 2019

mishmosh commented Jan 6, 2020

ZenGround0 commented Jul 6, 2020

ZenGround0 commented Jun 7, 2019 •

edited by pooja

Loading

ZenGround0 Jun 7, 2019 •

edited

Loading