Retry port forwarding when we see forwarding-related errors from kubectl

[nkubala]

sometimes kubectl port-forward will throw errors that look like
error: unable to forward port because pod is not running. Current status=Pending
and sometimes it will throw errors that look like
error forwarding port....

now, we'll just retry anytime we see an error in the logs from kubectl port-forward.

[tejal29]

hmm, lets hold off this change.

i tried this on my branch. I see a lot of retry loops.

DEBU[0004] Running command: [kubectl logs --since=2s -f gke-loadbalancer-694c964ddb-dghm7 -c gke-container --namespace default] 
Port forwarded service/gke-loadbalancer from remote port 80 to local port 4503
Port forwarded service/gke-loadbalancer from remote port 80 to local port 4503
DEBU[0005] Found dependencies for dockerfile: [{main.go /go true}] 
DEBU[0009] port forwarding service-gke-loadbalancer-skaffoldkwq7l-80 terminated: exit status 1, output: error: timed out waiting for the condition 
INFO[0009] retrying kubectl port-forward due to error: error: timed out waiting for the condition 
DEBU[0009] Terminating port-forward service-gke-loadbalancer-skaffoldkwq7l-80 
Port forwarded service/gke-loadbalancer from remote port 80 to local port 4503
DEBU[0015] port forwarding service-gke-loadbalancer-skaffoldkwq7l-80 terminated: exit status 1, output: error: timed out waiting for the condition 
INFO[0015] retrying kubectl port-forward due to error: error: timed out waiting for the condition 
DEBU[0015] Terminating port-forward service-gke-loadbalancer-skaffoldkwq7l-80 
Port forwarded service/gke-loadbalancer from remote port 80 to local port 4503
DEBU[0021] port forwarding service-gke-loadbalancer-skaffoldkwq7l-80 terminated: exit status 1, output: error: timed out waiting for the condition 
INFO[0021] retrying kubectl port-forward due to error: error: timed out waiting for the condition 
DEBU[0021] Terminating port-forward service-gke-loadbalancer-skaffoldkwq7l-80 
Port forwarded service/gke-loadbalancer from remote port 80 to local port 4503
DEBU[0027] port forwarding service-gke-loadbalancer-skaffoldkwq7l-80 terminated: exit status 1, output: error: timed out waiting for the condition 
INFO[0027] retrying kubectl port-forward due to error: error: timed out waiting for the condition 
DEBU[0027] Terminating port-forward service-gke-loadbalancer-skaffoldkwq7l-80 
Port forwarded service/gke-loadbalancer from remote port 80 to local port 4503
DEBU[0033] port forwarding service-gke-loadbalancer-skaffoldkwq7l-80 terminated: exit status 1, output: error: timed out waiting for the condition 
INFO[0033] retrying kubectl port-forward due to error: error: timed out waiting for the condition 
DEBU[0033] Terminating port-forward service-gke-loadbalancer-skaffoldkwq7l-80 
Port forwarded service/gke-loadbalancer from remote port 80 to local port 4503
DEBU[0039] port forwarding service-gke-loadbalancer-skaffoldkwq7l-80 terminated: exit status 1, output: error: timed out waiting for the condition 
INFO[0039] retrying kubectl port-forward due to error: error: timed out waiting for the condition 
DEBU[0039] Terminating port-forward service-gke-loadbalancer-skaffoldkwq7l-80 
Port forwarded service/gke-loadbalancer from remote port 80 to local port 4503

[tejal29]

on the other hand,
skaffold run and kubectl port-forward works
e.g.

tejaldesai:gke_loadbalancer (gke_load_balancer_svc)$ skaffold run
Generating tags...
 - gcr.io/tejal-test/gke-loadbalancer -> gcr.io/tejal-test/gke-loadbalancer:v0.33.0-211-ga1efe8cc-dirty
Tags generated in 19.502869ms
Starting build...
Building [gcr.io/tejal-test/gke-loadbalancer]...
Sending build context to Docker daemon  3.072kB
Step 1/6 : FROM golang:1.12.6-alpine3.9 as builder
 ---> e04879bf1b7f
Step 2/6 : COPY main.go .
 ---> Using cache
 ---> 101258c63295
Step 3/6 : RUN go build -o /main .
 ---> Using cache
 ---> e6e88bb93e06
Step 4/6 : FROM alpine:3.9
 ---> 055936d39205
Step 5/6 : CMD ["./main"]
 ---> Using cache
 ---> 0722ec521819
Step 6/6 : COPY --from=builder /main .
 ---> Using cache
 ---> f143958b6395
Successfully built f143958b6395
Successfully tagged gcr.io/tejal-test/gke-loadbalancer:v0.33.0-211-ga1efe8cc-dirty
The push refers to repository [gcr.io/tejal-test/gke-loadbalancer]
fbb38a418513: Preparing
f1b5933fe4b5: Preparing
f1b5933fe4b5: Layer already exists
fbb38a418513: Layer already exists
v0.33.0-211-ga1efe8cc-dirty: digest: sha256:f434d699b13cebd3d3ba7b094f2dc743149668d0615cae19530d730008fe454e size: 739
Build complete in 2.246788057s
Starting test...
Test complete in 12.559µs
Starting deploy...
kubectl client version: 1.11+
kubectl version 1.12.0 or greater is recommended for use with Skaffold
deployment.apps/gke-loadbalancer unchanged
service/gke-loadbalancer unchanged
Deploy complete in 875.481443ms
You can also run [skaffold run --tail] to get the logs
tejaldesai:gke_loadbalancer (gke_load_balancer_svc)$ kubectl port-forward svc/gke-loadbalancer 4503:80
Forwarding from 127.0.0.1:4503 -> 3000
Forwarding from [::1]:4503 -> 3000

Handling connection for 4503
Handling connection for 4503


Handling connection for 4503

[nkubala]

ugh, kubectl port-forward logging is really horrible. it logs an error: timed out waiting for the condition when it's trying to validate the port, but this isn't really an "error" because it retries silently on its own. let's match on the two port-forwarding related error strings (that are actual errors) instead of just a blanket error.

[codecov]

Codecov Report

Merging #2566 into master will not change coverage.
The diff coverage is 0%.

Impacted Files	Coverage Δ
...affold/kubernetes/portforward/kubectl_forwarder.go	0% <0%> (ø)	⬆️