[CI] Update documents around the GCP runners (#367)

This PR expands the documentation around GCP and the K8 cluster running
linux/windows runners.

premerge/README.md

@@ -5,27 +5,9 @@ resources used to run the premerge checks. Currently, only Google employees
 with access to the GCP project where these checks are hosted are able to apply
 changes. Pull requests from anyone are still welcome.
 
-## Setup
+## Index
 
-- install terraform (https://developer.hashicorp.com/terraform/install?product_intent=terraform)
-- get the GCP tokens: `gcloud auth application-default login`
-- initialize terraform: `terraform init`
-
-To apply any changes to the cluster:
-- setup the cluster: `terraform apply`
-- terraform will list the list of proposed changes.
-- enter 'yes' when prompted.
-
-## Setting the cluster up for the first time
-
-```
-terraform apply -target google_container_node_pool.llvm_premerge_linux_service
-terraform apply -target google_container_node_pool.llvm_premerge_linux
-terraform apply -target google_container_node_pool.llvm_premerge_windows
-terraform apply
-```
-
-Setting the cluster up for the first time is more involved as there are certain
-resources where terraform is unable to handle explicit dependencies. This means
-that we have to set up the GKE cluster before we setup any of the Kubernetes
-resources as otherwise the Terraform Kubernetes provider will error out.
+- [Architecture overview](architecture.md)
+- [Cluster management](cluster-management.md)
+- [Monitoring](monitoring.md)
+- [Past issues](issues.md)

premerge/architecture.md

@@ -0,0 +1,96 @@
+# LLVM Premerge infra - GCP runners
+
+This document describes how the GCP based presubmit infra is working, and
+explains common maintenance actions.
+
+## Overview
+
+Presubmit tests are using GitHub workflows. Executing GitHub workflows can be
+done in two ways:
+ - using GitHub provided runners.
+ - using self-hosted runners.
+
+GitHub provided runners are not very powerful, and have limitations, but they
+are **FREE**.
+Self hosted runners are self-hosted, meaning they can be large virtual
+machines running on GCP, very powerful, but **expensive**.
+
+To balance cost/performance, we keep both types.
+ - simple jobs like `clang-format` shall run on GitHub runners.
+ - building & testing LLVM shall be done on self-hosted runners.
+
+LLVM has several flavor of self-hosted runners:
+ - libcxx runners.
+ - MacOS runners for HLSL managed by Microsoft.
+ - GCP windows/linux runners managed by Google.
+
+This document only focuses on Google's GCP hosted runners.
+
+Choosing on which runner a workflow runs is done in the workflow definition:
+
+```
+jobs:
+  my_job_name:
+    # Runs on expensive GCP VMs.
+    runs-on: llvm-premerge-linux-runners
+```
+
+Our self hosted runners come in two flavors:
+  - Linux
+  - Windows
+
+## GCP runners - Architecture overview
+
+Our runners are hosted on a GCP Kubernetes cluster, and use the [Action Runner Controller (ARC)](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/about-actions-runner-controller).
+The cluster has 3 pools:
+  - llvm-premerge-linux
+  - llvm-premerge-linux-service
+  - llvm-premerge-windows
+
+**llvm-premerge-linux-service** is a fixed pool, only used to host the
+services required to manage the premerge infra (controller, listeners,
+monitoring). Today, this pool has three `e2-highcpu-4` machine.
+
+**llvm-premerge-linux** is a auto-scaling pool with large `n2-standard-64`
+VMs. This pool runs the Linux workflows.
+
+**llvm-premerge-windows** is a auto-scaling pool with large `n2-standard-64`
+VMs. Similar to the Linux pool, but this time it runs Windows workflows.
+
+### Service pool: llvm-premerge-linux-service
+
+This pool runs all the services managing the presubmit infra.
+  - Action Runner Controller
+  - 1 listener for the Linux runners.
+  - 1 listener for the windows runners.
+  - Grafana Alloy to gather metrics.
+  - metrics container.
+
+The Action Runner Controller listens on the LLVM repository job queue.
+Individual jobs are then handled by the listeners.
+
+How a job is run:
+ - The controller informs GitHub the self-hosted runner set is live.
+ - A PR is uploaded on GitHub
+ - The listener finds a Linux job to run.
+ - The listener creates a new runner pod to be scheduled by Kubernetes.
+ - Kubernetes adds one instance to the Linux pool to schedule new pod.
+ - The runner starts executing on the new node.
+ - Once finished, the runner dies, meaning the pod dies.
+ - If the instance is not reused in the next 10 minutes, the autoscaler
+   will turn down the instance, freeing resources.
+
+### Worker pools : llvm-premerge-linux, llvm-premerge-windows
+
+To make sure each runner pod is scheduled on the correct pool (linux or
+windows, avoiding the service pool), we use labels and taints.
+
+The other constraints we define are the resource requirements. Without
+information, Kubernetes is allowed to schedule multiple pods on the instance.
+So if we do not enforce limits, the controller could schedule 2 runners on
+the same instance, forcing containers to share resources.
+
+Those bits are configures in the
+[linux runner configuration](linux_runners_values.yaml) and
+[windows runner configuration](windows_runners_values.yaml).
+

premerge/cluster-management.md

@@ -0,0 +1,100 @@
+# Cluster configuration
+
+The cluster is managed using Terraform. The main configuration is
+[main.tf](main.tf).
+
+---
+NOTE: As of today, only Googlers can administrate the cluster.
+---
+
+Terraform is a tool to automate infrastructure deployment. Basic usage is to
+change this configuration and to call `terraform apply` make the required
+changes.
+Terraform won't recreate the whole cluster from scratch every time, instead
+it tries to only apply the new changes. To do so, **Terraform needs a state**.
+
+**If you apply changes without this state, you might break the cluster.**
+
+The current configuration stores its state into a GCP bucket.
+
+
+## Accessing Google Cloud Console
+
+This web interface is the easiest way to get a quick look at the infra.
+
+---
+IMPORTANT: cluster state is managed with terraform. Please DO NOT change
+shapes/scaling, and other settings using the cloud console. Any change not
+done through terraform will be at best overridden by terraform, and in the
+worst case cause an inconsistent state.
+---
+
+The main part you want too look into is `Menu > Kubernetes Engine > Clusters`.
+
+Currently, we have 3 clusters:
+ - `llvm-premerge-checks`: the cluster hosting BuildKite Linux runners.
+ - `windows-cluster`: the cluster hosting BuildKite Windows runners.
+ - `llvm-premerge-prototype`: the cluster for those GCP hoster runners.
+
+Yes, it's called `prototype`, but that's the production cluster.
+We should rename it at some point.
+
+To add a VM to the cluster, the VM has to come from a `pool`. A `pool` is
+a group of nodes withing a cluster that all have the same configuration.
+
+For example:
+A pool can say it contains at most 10 nodes, each using the `c2d-highcpu-32`
+configuration (32 cores, 64GB ram).
+In addition, a pool can `autoscale` [docs](https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler).
+
+If you click on `llvm-premerge-prototype`, and go to the `Nodes` tab, you
+will see 3 node pools:
+- llvm-premerge-linux
+- llvm-premerge-linux-service
+- llvm-premerge-windows
+
+Definitions for each pool is in [Architecture overview](architecture.md).
+
+If you click on a pool, example `llvm-premerge-linux`, you will see one
+instance group, and maybe several nodes.
+
+Each created node must be attached to an instance group, which is used to
+manage a group of instances. Because we use automated autoscale, and we have
+a basic cluster, we have a single instance group per pool.
+
+Then, we have the nodes. If you are looking at the panel during off hours,
+you might see no nodes at all: when no presubmit is running, no VM is on.
+If you are looking at the panel at peak time, you should see 4 instances.
+(Today, autoscale is capped at 4 instances).
+
+If you click on a node, you'll see the CPU usage, memory usage, and can access
+the logs for each instance.
+
+As long as you don't click on actions like `Cordon`, `Edit`, `Delete`, etc,
+navigating the GCP panel should not cause any harm. So feel free to look
+around to familiarize yourself with the interface.
+
+## Setup
+
+- install terraform (https://developer.hashicorp.com/terraform/install?product_intent=terraform)
+- get the GCP tokens: `gcloud auth application-default login`
+- initialize terraform: `terraform init`
+
+To apply any changes to the cluster:
+- setup the cluster: `terraform apply`
+- terraform will list the list of proposed changes.
+- enter 'yes' when prompted.
+
+## Setting the cluster up for the first time
+
+```
+terraform apply -target google_container_node_pool.llvm_premerge_linux_service
+terraform apply -target google_container_node_pool.llvm_premerge_linux
+terraform apply -target google_container_node_pool.llvm_premerge_windows
+terraform apply
+```
+
+Setting the cluster up for the first time is more involved as there are certain
+resources where terraform is unable to handle explicit dependencies. This means
+that we have to set up the GKE cluster before we setup any of the Kubernetes
+resources as otherwise the Terraform Kubernetes provider will error out.

premerge/issues.md

@@ -0,0 +1,125 @@
+# Past Issues
+
+This document lists past issues that could be of interest if you encounter
+issues with the cluser/presubmit.
+
+## Workflows are failing: DNS resolution of github.com fails.
+
+### Date: 2025-01-27
+
+### Symptoms:
+
+We noticed GitHub jobs were failing, and the logs showed `hostname lookup
+failed for github.com`.
+
+### Investigation
+
+Initial steps were to check the github status page. No outages.
+Then, we checked internal incident page. No outages.
+
+Then we looked at the instance logs.
+When a node fails, GCP removes it, meaning the logs are not accessible from
+the node page anymore, but can be retrieved either in the global logs.
+
+Looking at the logs, we discovered other services failing to resolve
+hostname, like the metrics container.
+
+In Kubernetes, each cluster runs a `kube-dns` service/pod, which is used
+by other pods to do DNS requests.
+This pod was crashing.
+
+Looking at the node this pod was running on showed a RAM usage close to the
+VM limits.
+At the time, the service instances were running on `e2-small` VMs, which only
+have 2GB of RAM.
+In addition, we recently added more runner nodes by adding a new Windows pool.
+This meant cluster size increased. This caused the cluster management services
+to take more resources to run, and pushed us just above the 2GB limit.
+
+This causes the kube-dns service to be OOM killed, and then caused various DNS
+failures in the cluster.
+
+### Solution
+
+Change the shape of the service pool to be `e2-highcpu-4`, doubling the RAM
+and CPU budget. We also increased the pool size from 2 to 3.
+
+## LLVM dashboard graphs are empty for presubmits
+
+### Date: 2025-01-28
+
+### Symptoms
+
+The LLVM dashboard was showing empty graphs for the presubmit job runtime and
+queue time. Autoscaling graphs were still working.
+
+### Investigation
+
+The graphs were empty because no new metrics were received, but other GCP
+metrics were still showing.
+Our dashboard has multiple data source:
+ - the GCP cluster.
+ - the metrics container.
+
+Because we had GCP metrics, it meant the Grafana instance was working, and
+the Grafana Alloy component running in the cluster was also fine.
+
+It was probably the metrics container.
+We checked the heartbeat metric: `metrics_container_heartbeat`.
+This is a simple ping recorded every minutes by the container. If this
+metrics stops emitting, it means something is wrong with the job.
+This metric was still being recorded.
+
+A recent change was made to add the windows version of the premerge check.
+This caused the job name to change, and thus changed the recorded metric
+names from `llvm_premerge_checks_linux_run_time` to
+`llvm_premerge_checks_premerge_checks_linux_run_time`.
+
+### Solution
+
+Change the dashboards to read the new metric name instead of the previous
+name, allowing new data to be shown.
+SLO definitions and alerts also had to be adjusted to look at the new metrics.
+
+## LLVM dashboard graphs are empty for run/queue times
+
+### Date: 2025-01-10
+
+### Symptoms
+
+The LLVM dashboard was showing empty graphs for the presubmit job runtime and
+queue time. Autoscaling graphs were still working.
+
+### Investigation
+
+Grafana was still recording GCP metrics, but no new data coming from the
+metrics container.
+
+A quick look at the google cloud console showed the metrics container pod was
+crashing.
+Looking at the logs, we saw the script failed to connect to GitHub to get
+the workflow status. Reason was a bad GitHub token.
+
+Because we have no admin access to the GitHub admin organization, we cannot
+emit LLVM owned tokens. A Googler had used its personal account to setup a
+PAT token. This token expired in December, causing the metrics container
+to fail since.
+
+### Solution
+
+Another Googler generated a new token, and replaced it in `Menu > Security > Secrets Manager > llvm-premerge-github-pat`.
+Note: this secret is in the general secret manager, not in `Kubernetes Engine > Secrets & ConfigMaps`.
+
+Once the secret updated, the metrics container had to be restarted:
+- `Menu > Kubernetes Engine > Workflows`
+- select `metrics`.
+- click `Rolling update`
+- set all thresholds to `100%`, the click update.
+
+This will allow GCP to delete the only metrics container pod, and recreate it
+using the new secret value.
+Because we have a single metrics container instance running, we have to but
+all thresholds to `100%`.
+
+In addition, we added a heartbeat metric to the container, and Grafana
+alerting to make sure we detect this kind of failure early.

premerge/monitoring.md

@@ -0,0 +1,20 @@
+# Monitoring
+
+Presubmit monitoring is provided by Grafana.
+The dashboard link is [https://llvm.grafana.net/dashboards](https://llvm.grafana.net/dashboards).
+
+Grafana pulls its data from 2 sources: the GCP Kubernetes cluster & GitHub.
+Grafana instance access is restricted, but there is a publicly visible dashboard:
+- [Public dashboard](https://llvm.grafana.net/public-dashboards/21c6e0a7cdd14651a90e118df46be4cc)
+
+## GCP monitoring
+
+Cluster metrics are gathered through Grafana alloy.
+This service is deployed using Helm, as described [HERE](main.tf)
+
+## Github monitoring
+
+Github CI queue and job status is fetched using a custom script which pushes
+metrics to grafana.
+The script itself lives in the llvm-project repository: [LINK](https://github.com/llvm/llvm-project/blob/main/.ci/metrics/metrics.py).
+The deployment configuration if in the [metrics_deployment file](metrics_deployment.yaml).