data set management

[reggev]

Description

This feature will enable managing data sources and applying them to pipelines

Includes

• a user can create a new data source entry on the app by uploading a new file and naming it
• when creating a new pipeline, the user can select a data source by its name
• a list of data sources for editing and deleting data sources
• A CRUD API on the API-server
• UI form for creating data sources
• a table for managing the collection
• update the adapters for the different languages to support this feature
• add the relevant data to the polling object the UI uses
• The data set's metadata is stored on MongoDB?

TODO:

• update package @hkube/parsers.
• Update package @hkube/storage-manager.
• Add new package @hkube/db.
• Update service Api-Server.
• Update service Pipeline-Driver.
• Run the full test case.
• Implement UI.

API:

Add new datasource

POST /datasource

• Name: 'model/binyanim'
• Description
• File: any file (if zip or .gz need to extract)