Download server for Velero client

[Lyndon-Li]

Many Velero client CLIs need to download data:

velero backup download
velero describe
velero backup logs
...

At present, these data are stored in the object store, so the client connect to the object store directly and download the data.
However, this has some drawbacks:

1. This assumes the target storing the data is always object store. In future, Velero may save the data elsewhere, i.e., NFS, or through Unified Repo, for which, the storage is transparent
2. This assumes the target storing the data is always accessible from the client side. This is not always true, especially in the on-premise environments

Therefore, we need to create a mechanism that get the data from the Velero server on behave of the client and deliver them to the client.

[ywk253100]

Maybe we can use the API Aggregation layer to achieve this.

[reasonerjt]

This essentially is decoupling downloadrequest from object storage.
IMO we need to decide on the priority and work on a design for this.

Setting it as a defer-candidate before we agree on the priority.

[kaovilai]

This would also solves the client having to specify --cacert or --insecure seperately.. although one could dynamically get these values from BSL today as well.

[kaovilai]

If we can do download server for object store velero plugins as well, it would solve the concern around local-volume-provider plugin, and enable other object stores like s3 with internal ip in s3url to work for clients as well.

This would probably be better approach than trying to re-implement NFS work internally and making this download server work exclusive to NFS work.

[Lyndon-Li]

If we want to make a built-in download server in Velero, I think it is not a good idea to have it storage specific, e.g., specific for object store or NFS. Instead, we should build a generic download server which talks to Velero server modules to get the data, the download server itself doesn't need to know the storage type. Reasons:

1. By building the server, we have completed most of the hard work and it is not difficult to make it visit both object store and NFS. Having them going in different ways makes the workflow complex and the troubleshooting difficult
2. We should decouple the download server's upper level request processing logics from the data accessing modules. It is a good architecture behavior and data accessing modules could be reused. So ideally, the upper level logics call Velero server modules to get the data, the download server itself doesn't need to be storage specific. This is helpful for future evolvement, e.g., we have a future plan to move all the backup data and metadata to unified repo managed storages, so that we are able to support advanced features. Then the download server is not able to probe the storage type/visit the storage on its own, but use Unified Repo API to get the data.

Conclusively:

• The download server should not be storage specific
• There should be two layers in the download server --- the request processing layer and the data accessing layer. The data accessing layer is shared throughout Velero

[kaovilai]

Currently debating approaches

Comparison

Wanted Features	1. API Aggregation Layer	2. Ingress
Support Multiple Velero in one cluster	❌	✅
Support multiple storage locations	✅	✅
K8S Style API	✅	❌
Authentication Built-in	✅	❌ has to parse TokenReview and SubjectAccessReview
Client works without resolvable DNS/IP if kubectl works	✅	❌ could avoid with kubectl port-forward-like approach?
Does not require ingress controller	✅	❌
Does not require NodePort, ClusterIP, LoadBalancer	✅	❌
Does not require apiserver specific flags	❌	✅

For up to date see: https://github.com/kaovilai/velero/blob/design-download-endpoint-for-client/design/velero-client-download-apiserver_design.md#comparison
Review PR at #7344