Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add docs around HA support for the pipeline controller #3418

Merged
merged 1 commit into from
Nov 5, 2020
+76 −0
Merged

Add docs around HA support for the pipeline controller #3418

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 72 additions & 0 deletions docs/developers/enabling-ha.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# HA support for Tekton Pipeline controllers

- [Overview](#overview)
- [Configuring HA](#configuring-ha)
- [Configuring the controller replicas](#configuring-the-controller-replicas)
- [Configuring the leader election](#configuring-the-leader-election)
- [Disabling leader election](#disabling-leader-election)
- [Use the disable-ha flag](#use-the-disable-ha-flag)
- [Scale down your replicas](#scale-down-your-replicas)

## Overview

---
This document is aimed at helping maintainers/developers when configuring High Availability(HA) support for the Tekton Pipeline [controller deployment](../../config/controller.yaml).

HA support allows components to remain operational when a disruption occurs. This is achieved by following an active/active model, where all replicas of the Tekton controller can receive workload. In this HA approach the reconcile space is distributed across buckets, where each replica owns a subset of those buckets and can process the load if the given replica is the leader of that bucket.

By default HA is enabled in the Tekton pipelines controller.

## Configuring HA

---
In order to achieve HA, the number of replicas for the Tekton Pipeline controller should be greater than one. This allows other instance(_s_) to take over in case of any disruption on the current active controller.

### Configuring the controller replicas

You can modify the replicas number in the [controller deployment](../../config/controller.yaml) under `spec.replicas` or apply an update to a running deployment:

```sh
kubectl -n tekton-pipelines scale deployment tekton-pipelines-controller --replicas=3
qu1queee marked this conversation as resolved.
Show resolved Hide resolved
```

### Configuring the Leader Election

The leader election can be configured via the [config-leader-election.yaml](../../config/config-leader-election.yaml). The configmap defines the following parameters:

| Parameter | Default |
| -------------------- | -------- |
| `data.resourceLock` | "leases" |
| `data.leaseDuration` | 15s |
| `data.renewDeadline` | 10s |
| `data.retryPeriod` | 2s |
Comment on lines +37 to +42
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mattmoor since this is inherited from knative/pkg, is there some documentation from that package that we can point to in here?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like that the key bucket is supported as well as I mentioned here: #3404, although they seem to be two different models of leadership election. As per my experiments, when the statefulset based leader election is enabled all instances become active. Perhaps @mattmoor can elaborate a litle about that.


## Disabling leader election

---

If HA is not required and running a single instance of the Tekton Pipeline controller is enough, there are two alternatives:

### Use the disable-ha flag

You can modify the [controller deployment](../../config/controller.yaml), by specifying in the `tekton-pipelines-controller` container the `disable-ha` flag. For example:

```yaml
spec:
serviceAccountName: tekton-pipelines-controller
containers:
- name: tekton-pipelines-controller
...
args: [
# Other flags defined here...
"-disable-ha=true"
]
```

by setting `disable-ha` to `true`, HA will be disable in the controllers.

**Note**: Please consider that when disabling HA and keeping multiple replicas of the [controller deployment](../../config/controller.yaml), each replica will act as an independent controller, leading to an unwanted behaviour when creating resources(e.g. `TaskRuns`, etc).

### Scale down your replicas

Although HA is enable by default, if your [controller deployment](../../config/controller.yaml) replicas are set to one, there would be no High Availability in the scenario where the running instance is deleted or fails. Therefore having a single replica even with HA enable, does not ensure high availability for the controller.
4 changes: 4 additions & 0 deletions docs/install.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -349,6 +349,10 @@ data:
disable-working-directory-overwrite: "true" # Tekton will not override the working directory for individual Steps.
```

### Customizing High Availability for the Pipelines Controller

To customize the behavior of HA for the Tekton Pipelines controller, please refer to the related [documentation](developers/enabling-ha.md).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the link is broken here, has extra /, link must point to developers/enabling-ha.md

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

## Creating a custom release of Tekton Pipelines

You can create a custom release of Tekton Pipelines by following and customizing the steps in [Creating an official release](https://github.com/tektoncd/pipeline/blob/master/tekton/README.md#create-an-official-release). For example, you might want to customize the container images built and used by Tekton Pipelines.
Expand Down