Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Design alternative Task Implementations inside a pipeline #215

Closed
dlorenc opened this issue Nov 2, 2018 · 10 comments
Closed

Design alternative Task Implementations inside a pipeline #215

dlorenc opened this issue Nov 2, 2018 · 10 comments
Assignees
Labels
area/roadmap Issues that are part of the project (or organization) roadmap (usually an epic) design This task is about creating and discussing a design lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale.

Comments

@dlorenc
Copy link
Contributor

dlorenc commented Nov 2, 2018

Right now A Pipeline is a graph of Tasks, where Tasks are coupled to a Build, which run as a Pod. We'll eventually need to support alternative implementations for the Build/Task execution for cases like:

  • Builds that cannot be containerized for some reason
  • Builds that must run on a specific piece of hardware (Mac/iOS)
  • Manual approval steps where a busy-looping container would be wasteful

< other use cases here >

@dlorenc
Copy link
Contributor Author

dlorenc commented Nov 2, 2018

cc @bobcatfish @abayer

@tanner-bruce
Copy link

Builds that must run on a specific piece of hardware (Mac/iOS)

knative/build#236 could be one possible solution

@bobcatfish bobcatfish added the design This task is about creating and discussing a design label Nov 6, 2018
@dlorenc
Copy link
Contributor Author

dlorenc commented Apr 23, 2019

Ref #781

@mumoshu
Copy link

mumoshu commented Jun 27, 2019

I have following use-cases to add :)

  • Runs Helmfile against the specified source to deploy a module of K8s apps
  • Runs any command in an existing pod i.e. kubectl exec, without bringing up a pod dedicated for running kubectl exec itself.
  • Sends a HTTP request to the specified endpoint, without creating a pod dedicated for running e.g. curl, and without maintaining the code in-tree.

For those, I'd be happy if Tekton provides us API to build and integrate a "custom task controller" in golang, where we have freedom to implement any logics to run the task in the controller.

I think we have three ways to achieve it in general:

  1. Add something like a CRD for Tekton Tasks
  2. Add a dedicated, Tekton-defined CRD for custom tasks
  3. Enhance the existing Task type to allow specifying the controller handles tasks, by controller
    name.

Add something like a CRD for Tekton Tasks

This can possibly be composed of:

  • Add tekton controllers an ability to handle any API resources in as unstructured. The user may name the CRD freely, like MyCustomTask
  • Enhance PipelineRun and TaskRun types to accept settings to create any K8s resources instead of tasks

One possible question here is - how Tekton fetches "logs" for the specific unstructured K8s resource?

For that, I believe we need at least a common interface between Tekton and the controller reconciling the K8s resources, maybe something like external_agent_logs in deck.

Add a dedicated, Tekton-defined CRD for custom tasks

Assuming we can use/enhance the existing Task type to cover use-cases involving pods, we'd only need a subset of Task removing pod/container-centric parts.

So it can basically be the same as tekton's Task except that steps can be unstructured(=defined by out-of-tree controllers) from the perspective of tekton's builtin controllers.

The same question about the common interface for fetching logs applies to this as well.

Enhance the existing Task type to allow specifying the controller handles tasks

This would add controllerName: mytaskcontroller to the Task API type, so that Tekton's builtin task controller is considered the default controller, which is used only when a task misses controllerName at all.

In case the custom task controller doesn't support steps, it can skip reading it. The controller would read at least inputs to parameterize the custom task though.

Tekton may provide a framework similar to controller-runtime to help devs building out-of-tree Tekton task controller. It should require at least controllerName as the parameter that is used to filter which tasks to reconcile.

@bobcatfish
Copy link
Collaborator

@imjasonh has been working on a design for custom tasks in tekton pipelines

@bobcatfish
Copy link
Collaborator

/assign @imjasonh

@bobcatfish bobcatfish added the area/roadmap Issues that are part of the project (or organization) roadmap (usually an epic) label Aug 24, 2020
@vdemeester
Copy link
Member

@bobcatfish @imjasonh should this be considered as done as "custom tasks" is a thing now ?

@tekton-robot
Copy link
Collaborator

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

@tekton-robot tekton-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 29, 2020
@vdemeester
Copy link
Member

With Custom tasks being a thing, I think this can be closed. I'll go ahead and close but if you feel it shouldn't be closed, please comment here and I'll re-open 👼🏼
/close

@tekton-robot
Copy link
Collaborator

@vdemeester: Closing this issue.

In response to this:

With Custom tasks being a thing, I think this can be closed. I'll go ahead and close but if you feel it shouldn't be closed, please comment here and I'll re-open 👼🏼
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/roadmap Issues that are part of the project (or organization) roadmap (usually an epic) design This task is about creating and discussing a design lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale.
Projects
None yet
Development

No branches or pull requests

7 participants