Sector storage groups

[magik6k]

This PR adds two new optional fields to sectorstore.json:

• Groups []string - list of group names the storage path belongs to.
• AllowTo []string - list of group names to which sectors can be fetched to from this storage path.

Those two should allow for creating worker groups, and avoiding unnecesarily moving data between multi-purpose workers. For example in the following setup:

• Storage miner
  • Path with Groups: ["example-storage-group-1"]
• Worker 1 (PC1, PC2):
  • Path with Groups: ["example-seal-group-1"], AllowTo: ["example-seal-group-1"]
• Worker 2 (PC1, PC2):
  • Path with Groups: ["example-seal-group-2"], AllowTo: ["example-seal-group-2"]
• Worker 3 (PC1):
  • Path with AllowTo: ["example-seal-group-1""]

Without storage groups, PC2 tasks on workers 1/2 could be scheduled with sector data from other workers, which would often waste bandwidth and occasionally block processing on fetching data.
With storage groups configured as above, sectors which had PC1 done on worker1 / worker2 will always execute PC2 on the same worker. Sectors from worker3 will only go to worker1 for PC2

Groups can be setup in two ways:

• For now storage paths, when using the storage attach --init command with the new --groups / --allow-to flags
• For existing storage paths - by modifying [path]/sectorstore.json, then restarting lotus-miner/worker

Groups:

{
  "ID": "74e1d667-7bc9-49bc-a9a6-0c30afd8684c",
  "Weight": 10,
  "CanSeal": false,
  "CanStore": true,
  "MaxStorage": 0,
  "Groups": ["storage0"]
}

AllowTo:

{
  "ID": "54f090bf-9fa9-4a47-9f67-0cc4f24c810e",
  "Weight": 10,
  "CanSeal": true,
  "CanStore": false,
  "AllowTo": ["storage0"]
}

TODO:

• CLI Utils
  • Show in storagelist
  • Storage attach --init flags
• Test on some miner setups
• Test with minerx miners

[codecov]

Codecov Report

Merging #7453 (b522901) into master (6c0374c) will increase coverage by 0.00%.
The diff coverage is 67.74%.

@@           Coverage Diff           @@
##           master    #7453   +/-   ##
=======================================
  Coverage   39.60%   39.60%           
=======================================
  Files         635      635           
  Lines       67551    67581   +30     
=======================================
+ Hits        26754    26766   +12     
- Misses      36203    36210    +7     
- Partials     4594     4605   +11     

Impacted Files	Coverage Δ
cmd/lotus-seal-worker/storage.go	0.00% <0.00%> (ø)
cmd/lotus-miner/storage.go	29.44% <14.28%> (-0.48%)	⬇️
extern/sector-storage/stores/local.go	59.16% <50.00%> (-0.11%)	⬇️
extern/sector-storage/stores/index.go	75.28% <100.00%> (+1.84%)	⬆️
chain/events/message_cache.go	87.50% <0.00%> (-12.50%)	⬇️
markets/loggers/loggers.go	89.28% <0.00%> (-10.72%)	⬇️
extern/sector-storage/manager_calltracker.go	57.70% <0.00%> (-4.85%)	⬇️
chain/stmgr/call.go	67.87% <0.00%> (-3.64%)	⬇️
blockstore/buffered.go	40.00% <0.00%> (-2.23%)	⬇️
chain/events/events_called.go	83.90% <0.00%> (-1.96%)	⬇️
... and 17 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6c0374c...b522901. Read the comment docs.

[jennijuju]

#7642