Add conditions to allow to wait for rollout pauses & completions

[M3t0r]

We have a workflow where we use make to rollout a new version, wait until the preview deployment is available, run some smoke tests, and then promote or rollback based on the result. Pretty basic. But for this to work we have to figure out when blue green rollout has paused.

I have found the kubectl wait command and looked a bit at how it works, waiting for conditions. But it seems it blocks the flow until the condition has it's status==true, which is not the case for paused BlueGreen rollouts. I didn't find a different way to wait for other metadata information in an object, so I had to implement my own loop and put it in a separate shell script.

I'd prefer to be able to use something like kubectl wait rollout $rollout --for=condition=PausedForPreview or something like that.

I haven't looked into the way canaries behave, but this would also be beneficial for canaries with (multiple) indefinite pause steps.

---

For reference:

Paused BlueGreen rollout (kubectl get rollout $rollout --output=json | jq .status):

{
  "HPAReplicas": 1,
  "availableReplicas": 1,
  "blueGreen": {
    "activeSelector": "6c4444d9b",
    "previewSelector": "cd564d8cc",
    "scaleUpPreviewCheckPoint": true
  },
  "canary": {},
  "conditions": [
    {
      "lastTransitionTime": "2020-01-10T14:47:39Z",
      "lastUpdateTime": "2020-01-10T14:47:39Z",
      "message": "Rollout has minimum availability",
      "reason": "AvailableReason",
      "status": "True",
      "type": "Available"
    },
    {
      "lastTransitionTime": "2020-01-13T16:08:34Z",
      "lastUpdateTime": "2020-01-13T16:08:34Z",
      "message": "Rollout is paused",
      "reason": "RolloutPaused",
      "status": "Unknown",
      "type": "Progressing"
    }
  ],
  "controllerPause": true,
  "currentPodHash": "cd564d8cc",
  "observedGeneration": "6b658b7868",
  "pauseConditions": [
    {
      "reason": "BlueGreenPause",
      "startTime": "2020-01-13T16:08:34Z"
    }
  ],
  "readyReplicas": 2,
  "replicas": 2,
  "selector": "app=$rollout,rollouts-pod-template-hash=6c4444d9b",
  "updatedReplicas": 1
}

Fully deployed BlueGreen rollout:

{
  "HPAReplicas": 1,
  "availableReplicas": 1,
  "blueGreen": {
    "activeSelector": "cd564d8cc",
    "previewSelector": "cd564d8cc"
  },
  "canary": {},
  "conditions": [
    {
      "lastTransitionTime": "2020-01-10T14:47:39Z",
      "lastUpdateTime": "2020-01-10T14:47:39Z",
      "message": "Rollout has minimum availability",
      "reason": "AvailableReason",
      "status": "True",
      "type": "Available"
    },
    {
      "lastTransitionTime": "2020-01-13T16:09:55Z",
      "lastUpdateTime": "2020-01-13T16:09:55Z",
      "message": "ReplicaSet \"$rollout-cd564d8cc\" has successfully progressed.",
      "reason": "NewReplicaSetAvailable",
      "status": "True",
      "type": "Progressing"
    }
  ],
  "currentPodHash": "cd564d8cc",
  "observedGeneration": "6b658b7868",
  "readyReplicas": 2,
  "replicas": 2,
  "selector": "app=$rollout,rollouts-pod-template-hash=cd564d8cc",
  "updatedReplicas": 1
}

This is the script I currently use to wait for that condition:

#!/bin/bash

rolloutname=${1?No rollout name provided}
KUBECTL=${KUBECTL:-kubectl}
KUBECTLFLAGS=${KUBECTLFLAGS}
JQ=${JQ:-jq}

set -ux

function is_paused() {
    # if you get 'jq: error (at <stdin>:384): Cannot iterate over null (null)'
    # that means that the object doesn't have a pauseConditions object, probably
    # because it isn't paused yet but still progressing
    $KUBECTL $KUBECTLFLAGS get rollout "$rolloutname" --output=json \
        | $JQ --exit-status '.status.pauseConditions[]|.reason=="BlueGreenPause"'
    return $?
}

function wait_for_pods_ready() {
    spec_hash=$(
        $KUBECTL $KUBECTLFLAGS get rollout "$rolloutname" --output=json \
        | $JQ --exit-status -r '.status.blueGreen.previewSelector'
    )

    $KUBECTL $KUBECTLFLAGS wait pods \
        --timeout=2m \
        --selector "rollouts-pod-template-hash=$spec_hash" \
        --for=condition=Ready
}


until is_paused; do
    sleep 2
done;

wait_for_pods_ready

[jessesuen]

This is a valid use case. In fact, we have something similar in our own pipelines, but since our deploys are done through Argo CD, we use:

argocd app sync my-app
argocd app wait my-app --suspended

It would be good to support this either using native kubectl wait, or the kubectl argo rollouts plugin

[cronik]

The condition for a "paused" rollout appears to be type Progressing with status Unknown

conditions:
  - lastTransitionTime: "2020-01-24T22:25:44Z"
    lastUpdateTime: "2020-01-24T22:25:44Z"
    message: Rollout has minimum availability
    reason: AvailableReason
    status: "True"
    type: Available
  - lastTransitionTime: "2020-01-25T00:40:45Z"
    lastUpdateTime: "2020-01-25T00:40:45Z"
    message: Rollout is paused
    reason: RolloutPaused
    status: Unknown
    type: Progressing

with that you can use the following kubectl wait command

kubectl wait --for=condition=Progressing=Unknown rollout/rollout-bluegreen
rollout.argoproj.io/rollout-bluegreen condition met

Might be worth having an explicit RolloutPaused condition to make the wait command more self-explanatory

kubectl wait --for=condition=RolloutPaused rollout/rollout-bluegreen

If you guys think it's worthy I can take a cut at implementing that.

[simonc-613]

Came here cause I'm looking for something similar.
We'll be using fully automated blue greens for the purpose of getting a clean traffic switch over.

We have a case where currently deployments are validated using helm upgrade --wait but as Helm doesn't support CRDs in --wait I need something to know when the Rollout is completed.

I'm in the process of promoting ArgoCD but the team who'll be using this initially won't be moving over.

Current thinking is to stick something like the below in an until/while loop

kubectl get rollouts.argoproj.io -n $NAMESPACE $ROLLOUT_NAME -o json | jq '.status.stableRS ==  .status.currentPodHash' --exit-status