Open Grant Proposal: Data Onboarding Manager: storage service manager

[Fatman13]

Open Grant Proposal: Data Onboarding Manager: storage service manager

Name of Project: Storage Service Manager for Venus Filecoin

Proposal Category: core-dev

Proposer: ipfsforce

(Optional) Technical Sponsor: n/a

Do you agree to open source all work you do on behalf of this RFP and dual-license under MIT, APACHE2, or GPL licenses?: Yes

Project Description

As deal-taking becomes a much more prominent task that SP handles on a daily basis, one of the issues that new SPs or even many veteran SPs are facing everyday when they on-board storage deals in scale is getting a clear picture of their deal-taking process to diagnose whatever has gone wrong in their pipeline. Many things could go wrong when on-boarding storage deals in large quantity, such as data transfer getting stuck, commP going wrong during AP, sealing pipeline not kicking in for deals etc. SPs have to navigate through these anomalies all the time and hustle to different sub-component of their storage system to set things right.


This is where ssm (storage-service-manager) for Venus Filecoin comes into play. We propose to build a manager service that could take in status information across different storage systems (nodes) and at the same time serve as a command center for these nodes to create a seamless operation experience so that SP wouldn't jump between individual services or nodes to issue commands or lookup status. ssm would be an entry interface to your storage system, which includes features such as replacing messages, statuses lookup, miners' info listing, storage deal management, owner/worker address management etc. Gernal architecture of how ssm fits into current SP storage system architecture would look like the following...

Value

There are many merits involved with using ssm to manage one's nodes across a secured private network. For example, if a SP operates multiple nodes within an IDC, it can then easily manage all its nodes through a ssm service as its main interface to interact with the system, which does not at all obstruct the running of different storage sub-systems. What's more, when SPs share a cloud chain service with each other, SPs will be able to replace messages, look up node status, check miner info through ssm service without the help of a chain service admin. Also ssm will be integrating a lot of the tooling functions which may not be appropriate for being included in other storage sub-components for advanced users to tinker with.

Risks

Implementations are expected to straight forward, but a bad architectural design may lead the project to a deadend. Before submitting the open grant proposal, the team had iterated on another alternative design which put ssm with "Cloud (Chain Service)" layer. This design was eventually vetoed for the following reasons...

• Further increases the complexity of operating a Venus chain service
• RPC endpoints of cluster and market sub-component would be exposed publicly, which raises security concerns
• Needs to build a new authentication mechanism for cluster

Deliverables

Three steps (architecture) of achieving SaaS with venus-cluster could be found here in details.

Development Roadmap

Mini Milestone	Area	Deliverable	Funding
A	Spec	Begin collecting feedbacks from community on what kind of ssm features they would like to see developed to ease their data-onboarding efforts	-
A	Spec	Initial design for ssm service with four major features chain management, storage deal management, storage power management, multi-sig management	4,000
B	Impl	Implementation of chain management functions of ssm as first MVP	5,000
B	Doc	Initial documentation on configurations and usages of the ssm system	3,000
C	Impl	Implementation of storage deal management functions of ssm	2,000
D	Impl	Implementation of storage power management functions of ssm	4,000
E	Impl	Implementation of multi-sig management functions of ssm	3,000
F	Spec	Release full documentation on the ssm system along with practical tutorials	3,000
G	Spec	Design for Web UI of ssm service and collect community feedbacks on more ssm use cases they would like to have after their initial tryouts	3,000
H	Impl	Implementation of web UI of ssm	6,000
I	Impl	Implementation of sealing thread management functions of ssm	4,000
-	-	Project Management: A dedicated project management budget will help coordinate work between different collaborators, as well as outreach to key stakeholders and key users	2,000
-	-	Total	$39,000

Delivery Roadmap


The development could be loosely broken down into three parts: 1) Design & MVP 2) Implementation and lastly 3) webUI & maintenance of the ssm system.

Design & MVP (Milestone 1)

This phase includes milestone A and B in the above deliverable table. The team will be collecting ideas from community, concieve the 1st design of ssm system, and lastly build a POC/MVP which includes chain management functionalities. Before ssm a Venus chain service operator will be responsible for all the status lookup and message replacement errands for SP using the chain service. Now with refactor of access control, an individual SP will be able to do chain related tasks by themselves in a secured and manageable fashion. Moreover, ssm itself will be a service which will have the capacity to streamline the management of multiple nodes in a private network.

Chian management features includes...

• actor lookups
  • head lookups
  • message lookups
  • message sending
  • miner_id specific message sending configurations
  • miner_id specific message replace

Implementation (Milestone 2)

This phase includes milestone C to F in the above deliverable table. The team will be continuing to collect ideas from community while implementing the rest of features of the ssm system. A list of features that ssm system will be incorporating are listed below…

• Storage deal management

  • storage deal lookup
  • storage deal metadata/status update

• Storage power management

  • miner info lookup
  • create miner
  • storage deal ask configurations
  • deadline information lookup
  • change of owner, worker, control address
  • funds management
  • sector information lookup
  • sector extend

• Multi-sig management

  • multi-sig address lookup
  • create new multi-sig address
  • delete multi-sig address
  • add new multi-sig signer
  • remove multi-sig signer
  • replace multi-sig signer
  • initiate a multi-sig proposal
  • lookup multi-sig proposal
  • cancel multi-sig proposal
  • pass multi-sig proposal

webUI & maintenance (Milestone 3)

This phase includes milestone G to I in the above deliverable table. The team will be continuing to collect ideas and feedbacks from community while iterating on the ssm system. Documentations and easy-to-follow tutorials will be produced to help push ssm system to be adopted by broader community members. We hope after we are done with this phase SPs will have the tools they need to serve as an unified interface to manage their storage providing nodes.

• all functions in milestone 1 & 2 enabled from Web UI
• sealing thread management functions
  • sealing thread lookups
  • sealing thread management
  • sealing thread abort / pause / start
  • configuration hot reload
  • sealing thread status update

Total Budget Requested

The total budget requests is $39,000. The breakdown of the budget is associated with the deliverables of each milestone, defined in the above table.


Maintenance and Upgrade Plans

The goal of the team is to support ssm system long term, which including continuously adding more critical features that community deemed worthy of adding. Therefore, easing the process of on-boarding large amount of data to the network and being the "command center" across different nodes in a secured and reliable way.


Team

Team Members

Force community engineering team

Team Member LinkedIn Profiles

Team Website

https://forcecommunity.io/

Relevant Experience

Force community has been an active contributor to Web3 ecosystem and Filecoin ecosystem in general. The engineering team from Force community has a track record of contributing code to Lotus as far back as Testnet and Space Race.

Team code repositories

https://github.com/ipfs-force-community

Additional information

Force community is committed to become a major contributor to Web3 infrastructure and we see Filecoin at the core of the big Web3 migration. We hope that we could fast track the realization of Web3 adoption by contributing our software development capacity to the course and join hand in hand with all other ecosystem developers around the globe through this historical journey!

[ErinOCon]

Hi @Fatman13, thank you for your proposal and for your patience with our review. We will not be proceeding with a grant at this time. Wishing you all the best as you continue to build!