prune-whitelist doesn't work for customresourcedefinitions

[thetillhoff]

What happened:

My kubernetes deployment pipeline has two (relevant) steps;

1. Apply CustomResourceDefinitions with kubectl apply -k ./crds
2. Apply Resources and CustomResources with kubectl apply --prune --all --prune-whitelist=apiextensions.k8s.io/v1 /customresourcedefinition -k ./

But for some reason, the second command will always prune the CustomResourceDefinitions anyway,

What you expected to happen:

The first step should apply all CustomResourceDefinitions that are located in ./crds/.
The second step should apply all resources and custom resources, prune the ones that don't exist in the folder any more, but not prune the CustomResourceDefinitions.

How to reproduce it (as minimally and precisely as possible):

My use case is cert-manager (v1.10.0), where I moved the CustomResourceDefinitions into the ./crds folder (taken from https://github.com/cert-manager/cert-manager/releases/download/v1.10.0/cert-manager.crds.yaml).
The custom resource I'm trying to deploy in the second step is a ClusterIssuer.

But I think generally any CustomResourceDefinition would be fine to verify this.

Anything else we need to know?:

• I'm using k3s.
• If there is a better way to deploy the CRDs alongside the CRs in one step, feel free to describe that as well.

Environment:

• Kubernetes client and server versions (use kubectl version):

Client Version: v1.24.6+k3s1
Kustomize Version: v4.5.4
Server Version: v1.24.6+k3s1

• Cloud provider or hardware configuration: k3s on amd64 bare metal
• OS (e.g: cat /etc/os-release): Debian 11

[k8s-ci-robot]

@thetillhoff: This issue is currently awaiting triage.

SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[brianpursley]

The second step should apply all resources and custom resources, prune the ones that don't exist in the folder any more, but not prune the CustomResourceDefinitions.

Prune whitelist works the opposite way. When specified, you are telling kubectl which resources can be pruned, so by specifying that, you are telling kubectl to ONLY prune CRDs.

You can specify the flag multiple times for multiple resource types (ie. --prune-whitelist core/v1/configmap --prune-whitelist core/v1/pod)

Or you can use label selectors to indicate which resources are eligible for pruning.

[eddiezane]

Please let us know if Brian's answer doesn't make sense.

/close
/kind support

[k8s-ci-robot]

@eddiezane: Closing this issue.

In response to this:

Please let us know if Brian's answer doesn't make sense.

/close
/kind support

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[thetillhoff]

Thank you Eddie for the reminder and thank you Brian for the explanation!

It makes sense to me and answers my question.

At the same it doesn't solve my problem 🥲
Could you please propose a way to apply CRDs and CRs in a pipeline (and removing them if they are removed from the repo)?
Maybe prune isn't the right direction, or is something like --blacklist planned in the future?