Monitoring Tekton Tasks and Pipelines with Elastic Observability

In a few minutes, you will have the following components up and running in your local Kubernetes cluster:

• Tekton v0.18.1 in the tekton-pipelines namespace

• Elastic Stack 7.10.0, Elasticsearch / Kibana / Beats (filebeat and metricbeat) deployed using Elastic Cloud on Kubernetes (ECK) in the elastic-system namespace

• Blog Post: https://discuss.elastic.co/t/dec-16th-2020-en-monitoring-tekton-tasks-and-pipelines-with-elastic-observability/257567

Prerequisites

• Kubernetes environment (Docker for Desktop, Minikube, GKE…​)

Helpers

If you don’t want to follow all the tutorial step by step, you can use the helpers scripts, make sure to be connected to your dev environment

# Install Tekton Pipelines / ECK / Elasticsearch / Kibana / Beats
$ ./helpers/set-up-env.sh

To add some data by executing some Tasks and Pipelines, run the following:

# Install some Tekton Tasks and Pipelines official examples
$ ./helpers/add-data.sh 0.18.1 default

The dashboard used in the demo is available at ./helpers/tekton-dashboard.ndjson

Use the script below to clean up your environment

# Install some Tekton Tasks and Pipelines official examples
$ ./helpers/clean-all.sh

Set up Tekton Pipelines and CLI

Install Tekton Pipelines CRD

# make sure you are using the right k8s context
$ kubectl config current-context
docker-for-desktop

# install Tekton Pipelines
$ kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.18.1/release.yaml

# Wait until all Tekton pods are ready
$ kubectl get pods -w -n tekton-pipelines
NAME                                           READY   STATUS    RESTARTS   AGE
tekton-pipelines-controller-77f74f5bcf-rbj8s   1/1     Running   0          1m
tekton-pipelines-webhook-f76c97965-5xkxq       1/1     Running   0          1m

Set up Tekton CLI

Find the best way to install it based on your local system by loogin at the official documentation:

Homebrew example

brew install tektoncd-cli

Check the versions

$ tkn version
Client version: 0.14.0
Pipeline version: v0.18.1

Get your first Tekton Task running!

Install Tekton Task from the catalog and execute it

# Install the git-clone Task from Tekton Catalog
$ tkn hub get task git-clone --version 0.2 |  kubectl apply -f -
task.tekton.dev/git-clone created

# Create a PVC to share data between tasks
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: go-source
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi
EOF
persistentvolumeclaim/go-source created

# Execute the Task to clone the repo
$ cat <<EOF | kubectl apply -f -
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: clone-my-code
spec:
  taskRef:
    name: git-clone
  workspaces:
  - name: output
    persistentVolumeClaim:
      claimName: go-source
  params:
  - name: url
    value: https://github.com/elastic/go-licenser.git
  - name: revision
    value: master
EOF
taskrun.tekton.dev/clone-my-code created

Everything is in place to execute a Task through the deployment of a first TaskRun that is going to invoke the Task:

# Execute a Task to clone the project
$ cat <<EOF | kubectl apply -f -
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: clone-my-code
spec:
  taskRef:
    name: git-clone
  workspaces:
  - name: output
    persistentVolumeClaim:
      claimName: go-source
  params:
  - name: url
    value: https://github.com/elastic/go-licenser.git
  - name: revision
    value: master
EOF
taskrun.tekton.dev/clone-my-code created

Check the result and logs

$ tkn taskruns list
NAME                                                  STARTED        DURATION     STATUS
clone-my-code                                         1 minute ago   9 seconds    Succeeded

$ tkn taskruns logs clone-my-code
...
[clone] + /ko-app/git-init -url https://github.com/elastic/go-licenser.git -revision v0.3.1 -refspec  -path /workspace/output/ '-sslVerify=true' '-submodules=true' -depth 1
[clone] {"level":"info","ts":1607989263.4070501,"caller":"git/git.go:165","msg":"Successfully cloned https://github.com/elastic/go-licenser.git @ 857b4969bc2f753ffb9eb3a885d01a59a9f22cdb (grafted, HEAD) in path /workspace/output/"}
[clone] ...

Let’s see how to use Elastic Observability to monitor any Tekton Tasks and Pipelines.

Set up Elastic Observability with Elastic Cloud on Kubernetes (ECK)

Install ECK

$ kubectl apply -f https://download.elastic.co/downloads/eck/1.3.1/all-in-one.yaml

Install Elastic Observability

# Deploy Elasticsearch and Kibana
$ kubectl apply -n elastic-system -f https://raw.githubusercontent.com/mgreau/tekton-pipelines-elastic-tutorials/master/config/eck/monitoring-es-kb.yaml

# Deploy Metricbeat and Filebeat
$ kubectl apply -n elastic-system -f https://raw.githubusercontent.com/mgreau/tekton-pipelines-elastic-tutorials/master/config/eck/monitoring-filebeat-metricbeat.yaml

When the set-up is done, you should have a all the Elastic components up and running in the elasic-system namespace:

# Check Elastic pods
$ kubectl get pods -n elastic-system
NAME                                    READY   STATUS    RESTARTS   AGE
elastic-operator-0                      1/1     Running   1          5m
elasticsearch-monitoring-es-default-0   1/1     Running   1          5m
filebeat-beat-filebeat-b6vlt            1/1     Running   7          5m
kibana-monitoring-kb-599698987-7cjqq    1/1     Running   1          5m
metricbeat-beat-metricbeat-fsz5p        1/1     Running   7          5m

Get the elastic user password needed to access the UI

$ echo $(kubectl get secret -n elastic-system elasticsearch-monitoring-es-elastic-user -o=jsonpath='{.data.elastic}' | base64 --decode)
XXXXXXXXXXXXXX

Make Elastic Kibana available

$ kubectl port-forward -n elastic-system svc/kibana-monitoring-kb-http 5601

Then you can access the following URL and use the credentials from above:

• https://localhost:5601/app/observability/overview

Note: the intent of this tutorial is to provide a development environment. To install a valide certificate, please refer to the official Elastic documentation.

Monitor Tasks and Pipelines with Elastic Observability

Run the tests Task and check the logs via the Logs UI

Install Tekton Tasks from the Catalog to build and test a golang project

# Install the generic golang-test Task from the catalog
$ tkn hub get task golang-test --version 0.1 |  kubectl apply -f -
task.tekton.dev/golang-test created

# Execute a Task to run the tests
$ cat <<EOF | kubectl apply -f -
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: test-my-code
spec:
  taskRef:
    name: golang-test
  workspaces:
  - name: source
    persistentVolumeClaim:
      claimName: go-source
  params:
  - name: package
    value: github.com/elastic/go-licenser
  - name: packages
    value: ./...
  - name: flags
    value: -timeout 10s -p 4 -race -cover
EOF
taskrun.tekton.dev/test-my-code created

Go back to the Metrics UI, there is now a Pod showing up named test-my-code in the default namespace. By clicking on this Pod, you can access the logs, you should see the following output:

Build the project

# build
$ cat <<EOF | kubectl apply -f -
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: build-my-code
spec:
  taskRef:
    name: golang-build
  workspaces:
  - name: source
    persistentVolumeClaim:
      claimName: go-source
  params:
  - name: package
    value: github.com/elastic/go-licenser
  - name: packages
    value: .
  - name: flags
    value: -o bin/go-licenser -ldflags="-X main.version=master-dev"
EOF
taskrun.tekton.dev/clone-my-code created

You can then filter the logs by the Task name, TaskRun name, and so on

Execute several Tasks and Pipelines and Monitor through a Tekton dashboard

Tekton Pipelines expose some metrics by default in Prometheus format. The metricbeat configuration provided earlier in this post has been initialized with the Prometheus module to automatically gather all these data:

Metricbeat Prometheus module

     - module: prometheus
        period: 10s
        hosts:
        - http://tekton-pipelines-controller.tekton-pipelines:9090
        metrics_path: /metrics

Therefore, your cluster is ready for having dashboards use these data. I have initialized one Tekton Dashboard that you can import using the Kibana Import feature. This dashboard is available at ./helpers/tekton-dashboard.ndjson

Execute multiple Tasks and Pipelines

# Run Tekton examples from the 0.18.1 git tag in the default namespace
$ ./helpers/add-data.sh 0.18.1 default

Now, open the Tekton Dashboard and watch the number of tasks running grow. You can also see which Tasks take the most time to execute.