-
Notifications
You must be signed in to change notification settings - Fork 385
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Discard unnecessary VolumeSnapshotContent updates to prevent rapid RPC calls #1009
Discard unnecessary VolumeSnapshotContent updates to prevent rapid RPC calls #1009
Conversation
/retest flake? |
/retest |
19a4874
to
365e41e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This PR looks great @ConnorJC3! manually tested using the EBS CSI driver by creating an invalid snapshot request to observe backoff behavior:
Without this patch:
kubectl logs ebs-csi-controller-54d895f474-bw4c8 -n kube-system | grep "CreateSnapshot"
Defaulted container "ebs-plugin" out of: ebs-plugin, csi-provisioner, csi-attacher, csi-snapshotter, csi-resizer, liveness-probe
I0217 00:48:04.536221 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-0a0a46ff2626acf04\" name:\"snapshot-edf397f6-b24e-479e-a447-3077b8beb1a5\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-edf397f6-b24e-479e-a447-3077b8beb1a5\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:48:04.650599 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-0a0a46ff2626acf04\" name:\"snapshot-edf397f6-b24e-479e-a447-3077b8beb1a5\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-edf397f6-b24e-479e-a447-3077b8beb1a5\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:48:04.746862 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-0a0a46ff2626acf04\" name:\"snapshot-edf397f6-b24e-479e-a447-3077b8beb1a5\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-edf397f6-b24e-479e-a447-3077b8beb1a5\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:48:04.828800 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-0a0a46ff2626acf04\" name:\"snapshot-edf397f6-b24e-479e-a447-3077b8beb1a5\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-edf397f6-b24e-479e-a447-3077b8beb1a5\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:48:04.911756 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-0a0a46ff2626acf04\" name:\"snapshot-edf397f6-b24e-479e-a447-3077b8beb1a5\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-edf397f6-b24e-479e-a447-3077b8beb1a5\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:48:04.989155 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-0a0a46ff2626acf04\" name:\"snapshot-edf397f6-b24e-479e-a447-3077b8beb1a5\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-edf397f6-b24e-479e-a447-3077b8beb1a5\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:48:05.057895 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-0a0a46ff2626acf04\" name:\"snapshot-edf397f6-b24e-479e-a447-3077b8beb1a5\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-edf397f6-b24e-479e-a447-3077b8beb1a5\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:48:05.161196 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-0a0a46ff2626acf04\" name:\"snapshot-edf397f6-b24e-479e-a447-3077b8beb1a5\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-edf397f6-b24e-479e-a447-3077b8beb1a5\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:48:05.258681 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-0a0a46ff2626acf04\" name:\"snapshot-edf397f6-b24e-479e-a447-3077b8beb1a5\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-edf397f6-b24e-479e-a447-3077b8beb1a5\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:48:05.336632 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-0a0a46ff2626acf04\" name:\"snapshot-edf397f6-b24e-479e-a447-3077b8beb1a5\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-edf397f6-b24e-479e-a447-3077b8beb1a5\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:48:05.424433 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-0a0a46ff2626acf04\" name:\"snapshot-edf397f6-b24e-479e-a447-3077b8beb1a5\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-edf397f6-b24e-479e-a447-3077b8beb1a5\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:48:05.509254 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-0a0a46ff2626acf04\" name:\"snapshot-edf397f6-b24e-479e-a447-3077b8beb1a5\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-edf397f6-b24e-479e-a447-3077b8beb1a5\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:48:05.582042 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-0a0a46ff2626acf04\" name:\"snapshot-edf397f6-b24e-479e-a447-3077b8beb1a5\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-edf397f6-b24e-479e-a447-3077b8beb1a5\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:48:05.681828 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-0a0a46ff2626acf04\" name:\"snapshot-edf397f6-b24e-479e-a447-3077b8beb1a5\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-edf397f6-b24e-479e-a447-3077b8beb1a5\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:48:05.757034 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-0a0a46ff2626acf04\" name:\"snapshot-edf397f6-b24e-479e-a447-3077b8beb1a5\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-edf397f6-b24e-479e-a447-3077b8beb1a5\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:48:05.829312 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-0a0a46ff2626acf04\" name:\"snapshot-edf397f6-b24e-479e-a447-3077b8beb1a5\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-edf397f6-b24e-479e-a447-3077b8beb1a5\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:48:05.968274 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-0a0a46ff2626acf04\" name:\"snapshot-edf397f6-b24e-479e-a447-3077b8beb1a5\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-edf397f6-b24e-479e-a447-3077b8beb1a5\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:48:06.036621 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-0a0a46ff2626acf04\" name:\"snapshot-edf397f6-b24e-479e-a447-3077b8beb1a5\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-edf397f6-b24e-479e-a447-3077b8beb1a5\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:48:06.113097 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-0a0a46ff2626acf04\" name:\"snapshot-edf397f6-b24e-479e-a447-3077b8beb1a5\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-edf397f6-b24e-479e-a447-3077b8beb1a5\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:48:06.187223 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-0a0a46ff2626acf04\" name:\"snapshot-edf397f6-b24e-479e-a447-3077b8beb1a5\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-edf397f6-b24e-479e-a447-3077b8beb1a5\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:48:06.250668 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-0a0a46ff2626acf04\" name:\"snapshot-edf397f6-b24e-479e-a447-3077b8beb1a5\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-edf397f6-b24e-479e-a447-3077b8beb1a5\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:48:06.323988 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-0a0a46ff2626acf04\" name:\"snapshot-edf397f6-b24e-479e-a447-3077b8beb1a5\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-edf397f6-b24e-479e-a447-3077b8beb1a5\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:48:06.385501 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-0a0a46ff2626acf04\" name:\"snapshot-edf397f6-b24e-479e-a447-3077b8beb1a5\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-edf397f6-b24e-479e-a447-3077b8beb1a5\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:48:06.503475 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-0a0a46ff2626acf04\" name:\"snapshot-edf397f6-b24e-479e-a447-3077b8beb1a5\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-edf397f6-b24e-479e-a447-3077b8beb1a5\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:48:06.594790 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-0a0a46ff2626acf04\" name:\"snapshot-edf397f6-b24e-479e-a447-3077b8beb1a5\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-edf397f6-b24e-479e-a447-3077b8beb1a5\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
...
With this patch:
kubectl logs ebs-csi-controller-cb74cdfbc-q5pdm -n kube-system | grep "CreateSnapshot"
Defaulted container "ebs-plugin" out of: ebs-plugin, csi-provisioner, csi-attacher, csi-snapshotter, csi-resizer, liveness-probe
I0217 00:41:38.559842 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-01405454f41560106\" name:\"snapshot-3e25e970-bf27-42e4-b259-4c7c41c46285\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-3e25e970-bf27-42e4-b259-4c7c41c46285\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:41:39.717545 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-01405454f41560106\" name:\"snapshot-3e25e970-bf27-42e4-b259-4c7c41c46285\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-3e25e970-bf27-42e4-b259-4c7c41c46285\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:41:41.798696 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-01405454f41560106\" name:\"snapshot-3e25e970-bf27-42e4-b259-4c7c41c46285\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-3e25e970-bf27-42e4-b259-4c7c41c46285\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:41:45.878465 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-01405454f41560106\" name:\"snapshot-3e25e970-bf27-42e4-b259-4c7c41c46285\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-3e25e970-bf27-42e4-b259-4c7c41c46285\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:41:53.948720 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-01405454f41560106\" name:\"snapshot-3e25e970-bf27-42e4-b259-4c7c41c46285\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-3e25e970-bf27-42e4-b259-4c7c41c46285\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:42:10.028229 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-01405454f41560106\" name:\"snapshot-3e25e970-bf27-42e4-b259-4c7c41c46285\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-3e25e970-bf27-42e4-b259-4c7c41c46285\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:42:42.100123 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-01405454f41560106\" name:\"snapshot-3e25e970-bf27-42e4-b259-4c7c41c46285\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-3e25e970-bf27-42e4-b259-4c7c41c46285\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
I0217 00:43:46.194127 1 controller.go:667] "CreateSnapshot: called" args="source_volume_id:\"vol-01405454f41560106\" name:\"snapshot-3e25e970-bf27-42e4-b259-4c7c41c46285\" parameters:<key:\"csi.storage.k8s.io/volumesnapshot/name\" value:\"ebs-volume-snapshot\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshot/namespace\" value:\"default\" > parameters:<key:\"csi.storage.k8s.io/volumesnapshotcontent/name\" value:\"snapcontent-3e25e970-bf27-42e4-b259-4c7c41c46285\" > parameters:<key:\"fastSnapshotRestoreAvailabilityZones\" value:\"us-west-2a, us-west-2b\" > "
built sidecar image for testing, available here: https://gallery.ecr.aws/m1r7q2k6/csi-snapshotter
Thanks
/lgtm
cc: @jsafrane
@torredil: changing LGTM is restricted to collaborators In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/assign @xing-yang |
365e41e
to
d79d0b8
Compare
d79d0b8
to
fc899f2
Compare
@xing-yang thanks for the review, all comments should be addressed and the PR should be ready for re-review now |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a nit. Looks good other than that.
…C calls Signed-off-by: Connor Catlett <[email protected]>
fc899f2
to
35579f2
Compare
/lgtm |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: ConnorJC3, xing-yang The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
What type of PR is this?
/kind bug
What this PR does / why we need it:
In many scenarios, the
csi-snapshotter
sidecar will enqueue aVolumeSnapshotContent
object in rapid succession, causing a flood of RPC requests to the attached CSI Driver.The most user-visible subset of this, when the CSI driver fails the request, has been reported and attempted to be fixed twice: #463 and #778. However, neither of the fixes completely fix the issue, which occurs even when the CSI driver does not return an error.
The most basic and common version of this issue is as follows:
VolumeSnapshotContent
object is enqueued (for example, because it is created)csi-snapshotter
will update the status of the contentexternal-snapshotter/pkg/sidecar-controller/snapshot_controller.go
Line 296 in ebf903a
csi-snapshotter
's own informer to fireexternal-snapshotter/pkg/sidecar-controller/snapshot_controller_base.go
Lines 118 to 127 in ebf903a
This loop will constantly spam the attached CSI driver with
CreateSnasphot
RPC calls until the snapshot actually is created (at which point, the sync will realize there's no action to take because the snapshot already exists, breaking out of the loop).You can very easily see this by upping the verbosity on the
csi-snapshotter
to 5, example log:How does this PR fix the bug?:
This PR adopts a similar approach to that used in other sidecars (for example the
external-attacher
's implementation of this idea: https://github.com/kubernetes-csi/external-attacher/blob/41cb23d3172818737b6573112219f443de3a6bf8/pkg/controller/controller.go#L282-L304). Changes in fields that we do not care about (and thus should not enqueue to sync for) are sanitized - specifically the following changes:csi-snapshotter
sidecar)The
VolumeSnapshotContent
is enqueued for sync only if there is still a difference after sanitization.Does this PR introduce a user-facing change?: