Deployment E2E tests failing in upgrade test jobs

[skriss]

@pwittrock you are listed as the test owner in test_owners.csv so directing this at you for now.

Most of the Deployment E2E tests are failing in the upgrade test jobs. See, for example:

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.4-container_vm-latest-upgrade-cluster/479

Common error message:

Expected an error to have occurred. Got:
: nil

Note that the 1.6 release upgrade test jobs are not set up yet, but I'm trying to get ahead of issues by looking at the jobs running off master. Thanks!

[0xmichalis]

@kubernetes/sig-apps-test-failures

[0xmichalis]

Why are we expecting an error to occur on every test?:)

[0xmichalis]

Is this test running 1.6 tests against a 1.4 cluster? This should explain the other deployment failure regarding the overlapping test.

[skriss]

It should actually be the opposite, i.e. running 1.4 tests against a 1.6 cluster (that was upgraded from a 1.4 cluster)

[0xmichalis]

The deployment overlap behavior has changed in 1.6 and is about to be dropped entirely, assuming #42175 gets merged. It was just bandage over a user error but that should not be possible in 1.6 with owner references. This means that the test is obsolete. I don't know what happens in this case. cc: @janetkuo

[0xmichalis]

I am talking about https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.4-container_vm-latest-upgrade-cluster/479#k8sio-deployment-overlapping-deployment-should-not-fight-with-each-other

[janetkuo]

Looking at the test, it expects get deployment to fail (right after calling deployment reaper) to make sure it's deleted https://github.com/kubernetes/kubernetes/blob/b440e9a9dbb30b/test/e2e/deployment.go#L206

[janetkuo]

Something to do with garbage collector?

[skriss]

@Kargakis @janetkuo can you clarify the status? To be clear, for 1.5->1.6 upgrades, there are 2 failing Deployment tests in the jobs that perform an upgrade then run 1.5 tests to check for compatibility. Are these issues that need to be addressed for the 1.6 release?

[k8s.io] Deployment deployment reaping should cascade to its replica sets and pods
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:62
Expected an error to have occurred. Got:
: nil
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:192

[k8s.io] Deployment overlapping deployment should not fight with each other
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:98
Failed to update the second deployment's overlapping annotation
Expected error:
<*errors.errorString | 0xc420388ca0>: {
s: "timed out waiting for the condition",
}
timed out waiting for the condition
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:1223

Note that both of these pass in the jobs that upgrade 1.5->1.6 and then run the 1.6 tests.

[skriss]

upgrade 1.5->1.6, run 1.5 tests: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.5-container_vm-1.6-upgrade-cluster/21

upgrade 1.5->1.6, run 1.6 tests: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.5-container_vm-1.6-upgrade-cluster-new/22

[0xmichalis]

upgrade 1.5->1.6, run 1.5 tests: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.5-container_vm-1.6-upgrade-cluster/21

The overlapping test is obsolete for 1.6 as the functionality has been totally rewritten.
@krmayankk can you have a look in the cascading deletion test as you introduced server side deletion in Deployments?

upgrade 1.5->1.6, run 1.6 tests: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.5-container_vm-1.6-upgrade-cluster-new/22

This is something that @enisoc is currently looking into - seems like the problem is that the upgrade suite is compiled with a 1.6 when it runs the 1.5 part of the upgrade.

[skriss]

@krmayankk please update by EOD today on whether the cascading deletion failure is a blocking issue for 1.6 or not; if it is, we need to get a PR in by tomorrow for the next beta/RC.

[0xmichalis]

@caesarxuchao can you also have a look? It seems related to the kubectl reaper not deleting the deployment, likely because it goes through the GC in the end?

[caesarxuchao]

The "[k8s.io] Deployment deployment reaping should cascade to its replica sets and pods" failure is because the deployment is not deleted yet when the 1.5 reaper.Stop() returns, because of the orphan finalizer. See #35676 (comment). 1.6 kubectl won't have this problem because of #40576 (thanks to @nikhiljindal).

I expect the upgrade test will fail all the deletion test for other controllers as well, after @enisoc's PRs that add controllerRef get merged.

I suggest that

1. we communicate this behavior change of old kubectl via CHANGELOG.
2. Cherrypick Updating kubectl to send delete requests with orphanDependents=false if --cascade is true #40576 to 1.5.

[janetkuo]

Filed #43041 to fix this test failure by updating 1.5 reapers (#40576 has changes we don't want, so filed a separate PR).

@skriss this doesn't block 1.6. The fix targets 1.5.

[0xmichalis]

@skriss can you give an update on this issue?

[skriss]

The deployment reaping issue appears to be fixed per https://k8s-testgrid.appspot.com/release-1.6-upgrade-skew#gke-container_vm-1.5-container_vm-1.6-upgrade-cluster&width=20

Thanks!