Move CSI snapshot KEP into directory and add an extension for tightening validation #1902

AndiLi99 · 2020-07-22T21:51:30Z

Create KEP extension for tightening validation on snapshot objects #1900

Some details are not fully fleshed out.

We need to carefully consider what validation will be tightened, and which fields will become immutable. What types of immutability will we use? Immutable on create, set or once 'valid'?
How exactly will the webhook server deploy with TLS? There is no standard boilerplate as far as I am aware of for creating webhook servers. Some suggestions include cert-manager. We need to consider whether TLS deployment is in scope for this feature. How will we ensure the certs don't interfere with existing cert management installed on existing clusters? How will we automate testing of the webhook?

The release process has been outlined in order to tackle backwards compatibility concerns.

cc @xing-yang @msau42 @yuxiangqian @mattcary

k8s-ci-robot · 2020-07-22T21:51:38Z

Welcome @AndiLi99!

It looks like this is your first PR to kubernetes/enhancements 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/enhancements has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2020-07-22T21:51:39Z

Hi @AndiLi99. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

msau42 · 2020-07-22T22:00:41Z

/ok-to-test

keps/sig-storage/177-volume-snapshot-validation/kep.yaml

keps/sig-storage/177-volume-snapshot-validation/tighten-validation-webhook-crd.md

msau42 · 2020-07-22T22:13:29Z

keps/sig-storage/177-volume-snapshot-validation/tighten-validation-webhook-crd.md

+1. The first phase is webhook-only, and will use ratcheting validation combined with a reconciliation method to delete or fix currently persisted objects which are invalid under the new (strict) validation rules.
+2. The second phase occurs once all invalid objects are cleared from the cluster. The CRD schema validation will be tightened and the webhook will stick around to enforce immutability until immutable fields come to CRDs. (or the crd upgrade could wait until immutable fields are available to do in one go)
+
+The phases come in separate releases to allow users / cluster admin the opportunity to clean their cluster of any invalid objects. More details are in the Risks and Mitigations section.


Is 1 release enough? I'm not quite sure where this falls under the kubernetes deprecation policy. This is changing the behavior of a beta API, but the deprecation policy mostly talks about removing entire API groups. cc @liggitt

Most of this tightening is like fixing bugs. The oneof restriction is already handled by the controller. So I think we should not wait for more than 1 release.

keps/sig-storage/177-volume-snapshot-validation/tighten-validation-webhook-crd.md

msau42 · 2020-07-22T22:19:38Z

@kubernetes/sig-storage-proposals

msau42 · 2020-07-22T22:20:37Z

cc @jsafrane @liggitt

yuxiangqian · 2020-07-22T22:22:25Z

cc @xing-yang

xing-yang · 2020-07-23T03:11:25Z

There's a validation error:

    --- FAIL: TestValidation/../../keps/sig-storage/177-volume-snapshot-validation/kep.yaml (0.00s)
        main_test.go:80: exit code was 1 and not 0. Output:
            ../../keps/sig-storage/177-volume-snapshot-validation/kep.yaml has an error: "error validating KEP metadata: \"prr-approvers\" must be one of (deads2k,johnbelamaric,wojtek-t) but it is a string: TBD"

AndiLi99 · 2020-07-23T15:33:57Z

@xing-yang who would be the prr reviewer for the original csi snapshot kep?

mattcary · 2020-07-23T16:53:25Z

@xing-yang who would be the prr reviewer for the original csi snapshot kep?

Looks like PRRs are new: #1620

There's a note in the changes to "Please reach out on the
#prod-readiness channel" for help & guidance.

I'll ask on there what to do.

xing-yang · 2020-07-23T19:11:47Z

The original snapshot KEP didn't go through the PRR process. You can pick one of (deads2k,johnbelamaric,wojtek-t) as PRR reviewer.

keps/sig-storage/177-volume-snapshot-validation/tighten-validation-webhook-crd.md

keps/sig-storage/177-volume-snapshot/tighten-validation-webhook-crd.md

xing-yang · 2020-07-24T04:01:02Z

keps/sig-storage/177-volume-snapshot/tighten-validation-webhook-crd.md

+
+## Proposal
+
+Tighten the validation on Volume Snapshot objects. The following fields will begin to enforce immutability: `VolumeSnapshot.Spec.Source` and `VolumeSnapshotContent.Spec.Source`. The following fields will begin to enforce `oneOf`: `VolumeSnapshot.Spec.Source` and `VolumeSnapshotContent.Spec.Source`). The following fields will begin to enforce non-empty strings: `spec.VolumeSnapshotClassName`. More details are in the Validating Scenarios section.


There's also spec.VolumeSnapshotRef

Added to the list with a note that it is immutable only after UID is set. I'm not sure if that behaviour is good though. I feel like we shall either make it fully immutable after creation, or not make it immutable at all. Thoughts?

Sorry, the VolumeSnapshotRef field says it's immutable after creation, is that inconsistent with the UID being set after creation? Also, I thought the controller created the whole VSC at once (but am I mistaken?)

In the dynamic snapshot case, the controller creates the entire VSC at once, you are correct.

In the pre-provisioned snapshot case, the CA must create the VSC. They must specify the name and namespace of the VS they wish to be bound to the VSC on the VSC.Spec.VolumeSnapshotRef field. If the VS does not exist yet there is no UID set, The controller after binding will set the UID in the VSC.Spec.VolumeSnapshotRef field. So the controller must be able to change that field after it is created to add set the UID one time.

This is a bit confusing but I'm not sure if anything can be done to change this behaviour.

cc @yuxiangqian to check if I got this behaviour right.

ok. Could you make a quick PR to update the comment in types.go so that it's consistent with what is actually the case?

what Andi described is accurate, I guess the semantic is more accurate if put this way: VSC.Spec.VolumeSnapshotRef is immutable (more specifically name, namespace, and UID fields) once bound.
for dynamic case, there is little ambiguity, all three fields will be set at once by controller.
for preprovisioned case, name/namespace are the users responsibility to set, while the UID is the controller's responsibility during binding (two stages instead of one compared to dynamic case). I think it's reasonable to allow updating for name/namespace fields BEFORE a content is bound to a VS, i.e., UID == "", for the pre-provisioned case. A potential use case could be:

CA creates a pre-provisioned VSC with VS-A name = "a", ns = "a-ns" (not bound yet)

CA realized they need the VSC to point to VS-B name = "b", ns = "b-ns"

CA updates the VSC to point to VS-B (if VSC has been bound(by checking UID == ""), it should be rejected by webhook, if not, webhook should allow?)

Yeah, I think those semantics make sense. But we shouldn't call the field just "immutable" in that case. I like making more precise by saying "immutable after binding" or something like that.

I'm making a quick PR to update the comment in types.go to say it's immutable after UID is set only.

keps/sig-storage/177-volume-snapshot/tighten-validation-webhook-crd.md

Fix typo Fix wrong date for beta of snapshot Change to reference release tag Add `VolumeSnapshotContent.spec.VolumeSnapshotRef` to list of immutable fields with note that it is immutable only after UID is set.

keps/sig-storage/177-volume-snapshot/tighten-validation-webhook-crd.md

mattcary · 2020-08-14T17:50:24Z

keps/sig-storage/177-volume-snapshot/tighten-validation-webhook-crd.md

+|  CREATE   |         spec.Source          |                                                                                       Exactly one of PersistentVolumeClaimName (Dynamic) or VolumeSnapshotContentName (Pre-provisioned) should be specified.                                                                                        |    400     |
+|  UPDATE   |         spec.Source          |                             Immutable, no updates allowed. If the user has specified an incorrect source, they must delete and remake the snapshot. The webhook validation server will not be able to guarantee that only incorrect sources are allowed to be updated.                              |    400     |
+|  CREATE   | spec.VolumeSnapshotClassName | Must not be the empty string. Can be unset (to use the default snapshot class, if it is set. If the default snapshot class is not set or there is more than 1 default class, then the hook will allow the creation but the snapshot will fail.), or set to a non-empty string (the snapshot class). |    400     |
+|  UPDATE   | spec.VolumeSnapshotClassName |                                    Same restrictions as CREATE. We won’t restrict updating by making this field immutable (only applying the same restrictions as creation) but this field should only be changed by those who know exactly what they are doing.                                    |    400     |


Can we add the reason for not restricting updates?

@yuxiangqian would know what specific scenarios are valid for changing the class name.

Can you add a TODO to resolve this by the time we implement the webhook? We should very clearly understand the use cases for supporting mutability of each field, and perhaps even document the use case in the CRD schema.

one scenario was the following:

user creates a VS which points to a non-existing volume snapshot class (fat-fingered or whatever reason), in this case the volume snapshot controller will fail the create a snapshot.

user updates the volume snapshot class to point to a correct one.
IMO it sounds like a reasonable mutation scenario?
If it's a mutable field, then it makes sense to me not to list this entry here?

What happens if a valid snapshot was created, but then the user changes the class to something different?

I guess there's a similar scenario where a user could delete and create a volumesnapshotclass with the same name, but completely different settings.

Added a TODO issue kubernetes-csi/external-snapshotter#362

…users can check if their cluster is okay to upgrade

msau42 · 2020-08-17T20:47:30Z

/approve
Awesome work Andy on tackling this difficult problem!

Will leave it to @xing-yang and @yuxiangqian for final review

yuxiangqian

overall LGTM. Great work Andi!

keps/sig-storage/177-volume-snapshot/tighten-validation-webhook-crd.md

yuxiangqian · 2020-08-17T21:07:06Z

keps/sig-storage/177-volume-snapshot/tighten-validation-webhook-crd.md

+|  CREATE   |         spec.Source          |                                                                                       Exactly one of PersistentVolumeClaimName (Dynamic) or VolumeSnapshotContentName (Pre-provisioned) should be specified.                                                                                        |    400     |
+|  UPDATE   |         spec.Source          |                             Immutable, no updates allowed. If the user has specified an incorrect source, they must delete and remake the snapshot. The webhook validation server will not be able to guarantee that only incorrect sources are allowed to be updated.                              |    400     |
+|  CREATE   | spec.VolumeSnapshotClassName | Must not be the empty string. Can be unset (to use the default snapshot class, if it is set. If the default snapshot class is not set or there is more than 1 default class, then the hook will allow the creation but the snapshot will fail.), or set to a non-empty string (the snapshot class). |    400     |
+|  UPDATE   | spec.VolumeSnapshotClassName |                                    Same restrictions as CREATE. We won’t restrict updating by making this field immutable (only applying the same restrictions as creation) but this field should only be changed by those who know exactly what they are doing.                                    |    400     |


one scenario was the following:

user creates a VS which points to a non-existing volume snapshot class (fat-fingered or whatever reason), in this case the volume snapshot controller will fail the create a snapshot.

user updates the volume snapshot class to point to a correct one.
IMO it sounds like a reasonable mutation scenario?
If it's a mutable field, then it makes sense to me not to list this entry here?

keps/sig-storage/177-volume-snapshot/tighten-validation-webhook-crd.md

xing-yang · 2020-08-17T23:27:30Z

keps/sig-storage/177-volume-snapshot/tighten-validation-webhook-crd.md

+Due to backwards compatibility concerns, the tightening will occur in three phases.
+
+1. The first phase is webhook-only, and will use [ratcheting validation](#backwards-compatibility). It will be the user's responsibility to clean up invalid objects which already existed before the webhook was enabled. Invalid objects are those which fail the new, stricter validation. The controller will not be able to automatically fix invalid objects, however it will apply a [label](#automatic-labelling-of-invalid-objects) to invalid objects so that users can easily locate them.
+2. The second phase can occur once all invalid objects are cleared from the cluster. It will be the cluster admin's responsibility to check and detect when it is safe to move to the second phase. The CRD schema validation will be tightened and the webhook will stick around to enforce immutability until immutable fields come to CRDs (Custom Resource Definition). This will be accompanied by a version change to make it clear the CRD is using different validation, however the storage version will be kept as `v1beta1` to ensure a [rollback](#rollback) is possible at phase 2.


What "version change" do you mean? Cut a different release?

Version change in the api of the object. So volume snapshot moves from v1beta1 to v1.

Can you clarify in the KEP that "version change" means snapshot API version moves from v1beta1 to v1?

It's not quite a "move" at this point. We're just making a new api version available at this stage, and changing our controllers to use the new api version.

What API version are you referring to? Is it "v1"?

Does phase 2 implies snapshot will go GA?

I reworded it to show that in phase 2 there will be a new snapshot version (v1). The plan is for phase 2 to be v1, implying GA.

If it is not GA ready, potentially the version could be v1beta2.

keps/sig-storage/177-volume-snapshot/tighten-validation-webhook-crd.md

yuxiangqian · 2020-08-18T03:15:54Z

LGTM
@liggitt @mattcary @xing-yang any more comments?

xing-yang · 2020-08-18T20:03:11Z

/lgtm
/approve

k8s-ci-robot · 2020-08-18T20:03:30Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AndiLi99, msau42, xing-yang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~keps/sig-storage/OWNERS~~ [msau42,xing-yang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

xing-yang · 2020-08-18T20:05:43Z

Thanks Andi!

AndiLi99 · 2020-08-18T22:39:22Z

Thank you everyone for all your help! Pleasure to work with you all.

Andi Li added 2 commits July 22, 2020 20:32

Convert CSI snapshot KEP into new style

cc3f0dd

Add tightening validation via webhook and crd schema KEP

b3b20d4

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jul 22, 2020

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 22, 2020

k8s-ci-robot requested review from saad-ali and xing-yang July 22, 2020 21:51

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 22, 2020

msau42 reviewed Jul 22, 2020

View reviewed changes

k8s-ci-robot added the kind/design Categorizes issue or PR as related to design. label Jul 22, 2020

Rename directory

4c8a898

jsafrane reviewed Jul 23, 2020

View reviewed changes

keps/sig-storage/177-volume-snapshot-validation/tighten-validation-webhook-crd.md Outdated Show resolved Hide resolved

Andi Li added 2 commits July 23, 2020 19:29

Fix some issues

ae1cfab

Fix list indent. Fix wording.

423cb9c

xing-yang reviewed Jul 24, 2020

View reviewed changes

Address comments

1917f94

Fix typo Fix wrong date for beta of snapshot Change to reference release tag Add `VolumeSnapshotContent.spec.VolumeSnapshotRef` to list of immutable fields with note that it is immutable only after UID is set.

saikat-royc reviewed Aug 12, 2020

View reviewed changes

keps/sig-storage/177-volume-snapshot/tighten-validation-webhook-crd.md Outdated Show resolved Hide resolved

saikat-royc reviewed Aug 12, 2020

View reviewed changes

keps/sig-storage/177-volume-snapshot/tighten-validation-webhook-crd.md Outdated Show resolved Hide resolved

saikat-royc reviewed Aug 12, 2020

View reviewed changes

keps/sig-storage/177-volume-snapshot/tighten-validation-webhook-crd.md Show resolved Hide resolved

Andi Li added 3 commits August 12, 2020 15:21

More comments

b62957f

More updates

3b2bedc

Add rollback proc

0f2c017

mattcary reviewed Aug 14, 2020

View reviewed changes

Andi Li added 4 commits August 14, 2020 17:44

Remove line about future changes to immutability. Add section on how …

a4984a6

…users can check if their cluster is okay to upgrade

Add a third phase in support of rollbacks in phase 2.

8dbf4ba

Fix table of contents

0cf8d23

recommend webhook same node as api server

4ec0b74

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 17, 2020

msau42 mentioned this pull request Aug 17, 2020

Allow ratcheting validation on custom resources kubernetes/kubernetes#94060

Closed

yuxiangqian reviewed Aug 17, 2020

View reviewed changes

AndiLi99 mentioned this pull request Aug 17, 2020

Add additional validation to Volume Snapshot objects #1900

Closed

Address comments

90dc842

AndiLi99 mentioned this pull request Aug 17, 2020

Document each field in snapshot objects as immutable / mutable with rationale kubernetes-csi/external-snapshotter#362

Closed

xing-yang reviewed Aug 17, 2020

View reviewed changes

AndiLi99 mentioned this pull request Aug 17, 2020

Add labels for invalid snapshot objects kubernetes-csi/external-snapshotter#363

Closed

Clarify version change

57b7a0d

k8s-ci-robot assigned xing-yang Aug 18, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 18, 2020

k8s-ci-robot merged commit fd1df62 into kubernetes:master Aug 18, 2020

k8s-ci-robot added this to the v1.19 milestone Aug 18, 2020


		## Proposal

		Tighten the validation on Volume Snapshot objects. The following fields will begin to enforce immutability: `VolumeSnapshot.Spec.Source` and `VolumeSnapshotContent.Spec.Source`. The following fields will begin to enforce `oneOf`: `VolumeSnapshot.Spec.Source` and `VolumeSnapshotContent.Spec.Source`). The following fields will begin to enforce non-empty strings: `spec.VolumeSnapshotClassName`. More details are in the Validating Scenarios section.

Move CSI snapshot KEP into directory and add an extension for tightening validation #1902

Move CSI snapshot KEP into directory and add an extension for tightening validation #1902

Conversation

AndiLi99 commented Jul 22, 2020

k8s-ci-robot commented Jul 22, 2020

k8s-ci-robot commented Jul 22, 2020

msau42 commented Jul 22, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msau42 commented Jul 22, 2020

msau42 commented Jul 22, 2020

yuxiangqian commented Jul 22, 2020

xing-yang commented Jul 23, 2020

AndiLi99 commented Jul 23, 2020

mattcary commented Jul 23, 2020

xing-yang commented Jul 23, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msau42 commented Aug 17, 2020

yuxiangqian left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuxiangqian commented Aug 18, 2020

xing-yang commented Aug 18, 2020

k8s-ci-robot commented Aug 18, 2020

xing-yang commented Aug 18, 2020

AndiLi99 commented Aug 18, 2020