Proto-docs: Compile-time and Runtime PodTemplates

[eapolinario]

Tracking issue

fixes #3385

Describe your changes

Documentation for Compile-time and Runtime PodTemplates feature.

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Note to reviewers

[hamersaw]

A few thoughts here:
(1) Should we be thinking of this as statically vs dynamically set rather than client-side vs server-side? In the realm of Flyte on k8s I think client / server boundaries are a little fuzzy, IMO a compile-time vs runtime differentiation is a little more understandable but maybe I'm too close to this.
(2) Is there anyway we can improve description on the order of application in this PR? With the original PodTemplate work that was a big concern that we had many issues on.

nit: stick to uniformity on PodTemplate terminology in different sentences we're using pod template or podtemplate.

[cosmicBboy]

Weird, sphinx thinks the subheader hierarchy is off https://github.com/flyteorg/flyte/actions/runs/4320063811/jobs/7539915673#step:5:121

Can you try using ^^^^^^^^^ instead of ------------?

Also, we should also add a new example in flytesnacks under the Kubernetes Pod integration
https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/pod/index.html

[hamersaw]

under the Kubernetes Pod integration

that's not the correct place as these will apply everywhere - ContainerTask, ShellTask, Dask / Ray plugins, etc

[cosmicBboy]

under the Kubernetes Pod integration

that's not the correct place as these will apply everywhere - ContainerTask, ShellTask, Dask / Ray plugins, etc

In that case, thoughts on where this should go?

[eapolinario]

under the Kubernetes Pod integration

that's not the correct place as these will apply everywhere - ContainerTask, ShellTask, Dask / Ray plugins, etc

In that case, thoughts on where this should go?

How about we add those in each relevant section? For example:

• ContainerTask
• ShellTask
• Dask
• Ray

@cosmicBboy , what do you think?

[eapolinario]

Here's how the little graph renders in the docs:

[hamersaw]

I think some of these examples may be redundant. Example 3 covers Example 1 and Example 2 right? OK to leave all as well.

[eapolinario]

Yeah, I just want to make it clear that we can mix and match (up to a point obviously, since one cannot use a default Runtime PodTemplate and a named one at the same time).

[hamersaw]

The Set the default-pod-template-name in FlytePropeller above this line is only required when using a default PodTemplate and not if it is set in the task decorator. Does it make sense to make this distinction?

[eapolinario]

I'm going to add a little blurb explaining the difference between the default Runtime PodTemplate and a named one.

[hamersaw]

We should explicitly say that the default-pod-template-name is used unless it is overridden by the pod-template-name in the task decorator. Only one may be applied. Not sure where is the best fit.

[eapolinario]

Good call.

[hamersaw]

Had to read in case (1) is active and FlytePropeller is building a Pod where (2) was used as part of the definition of the task, then only the values of the PodTemplate mentioned in (2) will be used to build the Pod. multiple times to understand. Might consider rewording? If not, OK too.