[Core feature] Supporting customized k8s client in various k8s plugins

[honnix]

Motivation: Why do you think this is important?

Organizations may have different Flyte setup, where one could set up centrally managed k8s clusters to run different types of jobs: batch, ML, etc.; while another setup could be different teams managing their own k8s clusters for different purposes.

It would be great to be able to configure the core k8s plugins to do inter-cluster communication. Plain pod is necessary.

Goal: What should the final outcome look like, ideally?

Users can configure core k8s plugin to do inter-cluster communication, essentially submitting custom resource to a specified k8s cluster instead of the one flytepropeller runs on.

Describe alternatives you've considered

Alternatively, a dedicated flytepropeller could be set up in the target cluster. However the team managing the external k8s cluster may not have sufficient knowledge to operate flytepropeller.

Another alternative is to make those core k8s plugins extensible so users can override the plugin registration process and provide their own k8s client via https://github.com/flyteorg/flyteplugins/blob/9e00636f9b2fcb84b5bc2c8dcece2b9b2bd0935e/go/tasks/pluginmachinery/k8s/plugin.go#L32. For example making https://github.com/flyteorg/flyteplugins/blob/master/go/tasks/plugins/k8s/ray/ray.go#L31-L40 public and https://github.com/flyteorg/flyteplugins/blob/master/go/tasks/plugins/k8s/ray/ray.go#L370 in a separated package, of course in a backward compatible way.

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[hamersaw]

Adding a bit of context here. The Flyte Flink plugin exposes configuration for the kubeclient. This is then passed to FlytePropeller instance during the registration process, where FlytePropeller registers the plugin to use the configured kubeclient.

In this paradigm the Flink plugin exposes specific kubeclient configuration; which, if exists overrides the default FlytePropeller configuration - allowing inter-cluster communication. This is advantageous in deployments where FlytePropeller is in one cluster, Ray may be executed in another cluster, Spark in yet another, etc.

The proposal is to add these configuration options to all k8s plugins. This involves adding kubeclient configuration to each individual plugin and including this during the FlytePropeller plugin registration process (similar to how this is currently performed in the Flink plugin). The result is configuration for each plugin that may look something like:

plugins:
  ray:
    # existing configuration omitted
    kube:
      endpoint: "localhost:30082"
  spark:
    # existing configuration omitted
    kube:
      endpoint: "localhost:30083"

[hamersaw]

This should be a relatively easy change. However, this deployment model is something that we don't suggest. This complicates things like secret management, inter-cluster management, exposing multiple failure domains, etc. For these reasons it's likely this will not get implemented unless it comes as a community contribution. We are very happy to help with anything, feel free to reach out.

[honnix]

Sorry for replying back.

I understand why this is not a recommended deployment model, but in practice, it is not always feasible to have a flytepropeller running in a k8s cluster. As examples, the cluster could be dedicated to run a specific type of load and due to security reason nothing else is allowed to run there; and the cluster could even be owned by a 3rd-party.

I think having this capability built into Flyte will help users integrate with their existing infrastructure more easily, and help build Flyte as a platform (things being integrated into Flyte, not the other way around). Please let me know what you think. Thank you for looking into this.

[github-actions]

Hello 👋, this issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will engage on it to decide if it is still applicable.
Thank you for your contribution and understanding! 🙏