Validate Design & Implement Tekton Logging Proposal

[ghost]

The Design Doc for this feature can be read and commented on here: Design Doc

Following on from #107 a design doc for logging in Tekton has been circulated with the working group and initial feedback has been gathered. It's now time to validate the proposal with potential early users and to implement it for review. This issue is an epic intended to encompass that work.

Tekton Logging Requirements

• Logs of TaskRuns can be accessed after a TaskRun’s pod has been garbage collected
• The method of persisting logs is pluggable. In the short term GCS support will be added and the interface for adding new storage types will be documented.
• A single PipelineRun’s logs, potentially spanning multiple TaskRuns, can be easily identified as related
• Tekton’s log uploading can be disabled by those with existing logs infra (e.g. a company already has a fluentd DaemonSet and doesn’t want Tekton’s built-in log streaming)
• The logs of a TaskRun or PipelineRun can still be monitored using kubectl, tkn, and the Tekton dashboard while a Run is executing.
• The logs are streamed to the persistence store. The entire log output must not be held in-memory or on the node’s disk during execution.

Tasks

• Modify the entrypoint binary to (optionally, configurably) send long lines to a sidecar
• Implement user-defined sidecars
• Build a configurable GCS log uploading sidecar for Tekton logs

[ghost]

Implementation of this design is halted in favor of existing solutions. Through conversations with a number of teams and some experimenting with fluentd / stackdriver it has become clear that Tekton does not need to invent its own log export solution. Fluentd is an existing tool that is highly configurable and well-documented. The labels that Tekton adds to its Tasks' Pods provide enough information to query for log output based on the Pipeline, PipelineRun, Task, or TaskRun that created the Pod and a container's name indicates which specific step a log entry was generated by.

User-defined sidecars in Tasks has been implemented and can be iterated on its own.

[arzarif]

Hi @sbwsg - what would this mean for historical data presented via Tekton dashboard? Presumably, the dashboard isn't going to be interfacing with whatever solution is being used for cluster-wide pod log collection. If that's correct, wouldn't that present a pretty big hit to the dashboard's utility?

[ghost]

The design here didn't dictate a specific persistent store for historical logs that would be universally available in all Tekton clusters. Even if this proposal were implemented the dashboard would have still needed to integrate with whatever persistent store an operator configured tekton to export out to. Really all we were doing in this design was pushing the configuration and burden of collection into Tekton instead of relying on the underlying platform.

iow I don't think the dashboard is affected one way or the other with the status of this proposal changing. Happy to discuss further though.

[arzarif]

Thanks for the clarification (unfortunately I can't access the design doc even though I'm on the mailing list). My initial read on this was that this specifically meant to enable historical logs within the dashboard. If that wasn't the intention, then I agree.

I wonder if the best approach is to just have the dashboard interface directly with some storage medium (S3, GCS, etc.) to persist logs it retrieves from underlying pods as they run. This would actually be more in line, I think, with the pattern Drone uses for log persistence. Do you have any insight into how the dashboard's design is being formulated?