feat: add wide cluster configuration for workload partitioning #1213

eggfoobar · 2022-08-09T03:49:07Z

initial commit of proposal

initial commit of proposal Signed-off-by: ehila <[email protected]> doc: updated wording and updated reviewers and approvers added clearity on risks and goals Signed-off-by: ehila <[email protected]>

enhancements/workload-partitioning/wide-availability-workload-partitioning.md

Signed-off-by: ehila <[email protected]>

enhancements/workload-partitioning/wide-availability-workload-partitioning.md

added variation for partition size added exlicit goal to maintain partition resize Signed-off-by: ehila <[email protected]>

added information for new approach to avoid race condition added information about how to limit users ability to turn off feature and install time configuration Signed-off-by: ehila <[email protected]>

enhancements/workload-partitioning/wide-availability-workload-partitioning.md

updated small wording changes gave more detail for upgrade path Signed-off-by: ehila <[email protected]>

enhancements/workload-partitioning/wide-availability-workload-partitioning.md

added context to capture proposal for machine config added more info for upgrades cleaned up older references Signed-off-by: ehila <[email protected]>

UPSTREAM: <carry>: openshift-kube-apiserver: enabled conversion gen for admission configs UPSTREAM: <carry>: openshift-kube-apiserver/admission: fix featuregates resource name UPSTREAM: <carry>: openshift-kube-apiserver/admission: add missing FeatureSets UPSTREAM: <carry>: openshift-kube-apiserver: use github.com/openshift/apiserver-library-go/pkg/labelselector UPSTREAM: <carry>: openshift authenticator: don't allow old-style tokens UPSTREAM: <carry>: oauth-authn: support sha256 prefixed tokens UPSTREAM: <carry>: oauth-token-authn: switch to sha256~ prefix UPSTREAM: <carry>: oauth-token-authn: add sha256~ support to bootstrap authenticator UPSTREAM: <drop>: remove the openshift authenticator from the apiserver In 4.8, we moved the authenticator to be configured via webhookTokenAuthenticators to an endpoint in the oauth-apiserver, this should now be safe to remove. UPSTREAM: <carry>: set ResourceQuotaValidationOptions to true When PodAffinityNamespaceSelector goes to beta or GA this might affect how our ClusterResourceQuota might work UPSTREAM: <carry>: simplify the authorizer patch to allow the flags to function UPSTREAM: <carry>: eliminate unnecessary closure in openshift configuration wiring UPSTREAM: <carry>: add crdvalidation for apiserver.spec.tlsSecurityProfile UPSTREAM: <carry>: openshift-kube-apiserver: Add custom resource validation for network spec UPSTREAM: <carry>: stop overriding flags that are explicitly set UPSTREAM: <carry>: add readyz check for openshift apiserver availability UPSTREAM: <carry>: wait for oauth-apiserver accessibility UPSTREAM: <carry>: provide a new admission plugin to mutate management pods CPUs requests The ManagementCPUOverride admission plugin replaces pod container CPU requests with a new management resource. It applies to all pods that: 1. are in an allowed namespace 2. and have the workload annotation. It also sets the new management resource request and limit and set resource annotation that CRI-O can recognize and apply the relevant changes. For more information, see - openshift/enhancements#703 Conditions for CPUs requests deletion: 1. The namespace should have allowed annotation "workload.openshift.io/allowed": "management" 2. The pod should have management annotation: "workload.openshift.io/management": "{"effect": "PreferredDuringScheduling"}" 3. All nodes under the cluster should have new management resource - "management.workload.openshift.io/cores" 4. The CPU request deletion will not change the pod QoS class UPSTREAM: <carry>: Does not prevent pod creation because of no nodes reason when it runs under the regular cluster Check the `cluster` infrastructure resource status to be sure that we run on top of a SNO cluster and in case if the pod runs on top of regular cluster, exit before node existence check. UPSTREAM: <carry>: do not mutate pods when it has a container with both CPU request and limit Removing the CPU request from the container that has a CPU limit will result in the defaulter to set the CPU request back equals to the CPU limit. UPSTREAM: <carry>: Reject the pod creation when we can not decide the cluster type It is possible a race condition between pod creation and the update of the infrastructure resource status with correct values under Status.ControlPlaneTopology and Status.InfrastructureTopology. UPSTREAM: <carry>: add CRD validation for dnses Add an admission plugin that validates the dnses.operator.openshift.io custom resource. For now, the plugin only validates the DNS pod node-placement parameters. This commit fixes bug 1967745. https://bugzilla.redhat.com/show_bug.cgi?id=1967745 * openshift-kube-apiserver/admission/customresourcevalidation/attributes.go (init): Install operatorv1 into supportedObjectsScheme. * openshift-kube-apiserver/admission/customresourcevalidation/customresourcevalidationregistration/cr_validation_registration.go (AllCustomResourceValidators, RegisterCustomResourceValidation): Register the new plugin. * openshift-kube-apiserver/admission/customresourcevalidation/dns/validate_dns.go: New file. (PluginName): New const. (Register): New function. Register the plugin. (toDNSV1): New function. Convert a runtime object to a versioned DNS. (dnsV1): New type to represent a runtime object that is validated as a versioned DNS. (ValidateCreate, ValidateUpdate, ValidateStatusUpdate): New methods. Implement the ObjectValidator interface, using the validateDNSSpecCreate and validateDNSSpecUpdate helpers. (validateDNSSpecCreate, validateDNSSpecUpdate): New functions. Validate a DNS, using the validateDNSSpec helper. (validateDNSSpec): New function. Validate the spec field of a DNS, using the validateDNSNodePlacement helper. (validateDNSNodePlacement): New function. Validate the node selector and tolerations in a DNS's node-placement parameters, using validateTolerations. (validateTolerations): New function. Validate a slice of corev1.Toleration. * openshift-kube-apiserver/admission/customresourcevalidation/dns/validate_dns_test.go: New file. (TestFailValidateDNSSpec): Verify that validateDNSSpec rejects invalid DNS specs. (TestSucceedValidateDNSSpec): Verify that validateDNSSpec accepts valid DNS specs. * vendor/*: Regenerate. UPSTREAM: <carry>: prevent the kubecontrollermanager service-ca from getting less secure UPSTREAM: <carry>: allow SCC to be disabled on a per-namespace basis UPSTREAM: <carry>: verify required http2 cipher suites In the Apiserver admission, we need to return an error if the required http2 cipher suites are missing from a custom tlsSecurityProfile. Currently, custom cipher suites missing ECDHE_RSA_WITH_AES_128_GCM_SHA256 or ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 result in invalid http2 Server configuration causing the apiservers to crash. See: go/x/net/http2.ConfigureServer for futher information. UPSTREAM: <carry>: drop the warning to use --keep-annotations When a user runs the `oc debug` command for the pod with the management resource, we will inform him that he should pass `--keep-annotations` parameter to the debug command. UPSTREAM: <carry>: admission/managementcpusoverride: cover the roll-back case During the upgrade and roll-back flow 4.7->4.8->4.7, the topology related fields under the infrastructure can be empty because the old API does not support them. The code will equal the empty infrastructure section with the current one. When the status has some other non-empty field, and topology fields are empty, we assume that the cluster currently passes via roll-back and not via the clean install. UPSTREAM: <carry>: Remove pod warning annotation when workload partitioning is disabled UPSTREAM: <carry>: use new access token inactivity timeout field. UPSTREAM: <carry>: apirequestcount validation UPSTREAM: <carry>: Added config node object validation for extreme latency profiles UPSTREAM: <carry>: Add Upstream validation in the DNS admission check patches UPSTREAM: <carry>: Make RestrictedEndpointsAdmission check NotReadyAddresses UPSTREAM: <carry>: Make RestrictedEndpointsAdmission restrict EndpointSlices as well Moved SkipSystemMasterAuthorizers to the authorizer. UPSTREAM: <carry>: Add validation plugin for CRD-based route parity. UPSTREAM: <carry>: Add host assignment plugin for CRD-based routes. UPSTREAM: <carry>: Apply shared defaulters to CRD-based routes. Signed-off-by: Artyom Lukianov <[email protected]> Signed-off-by: Damien Grisonnet <[email protected]> Signed-off-by: Swarup Ghosh <[email protected]> OpenShift-Rebase-Source: 932411e OpenShift-Rebase-Source: 1899555 OpenShift-Rebase-Source: 453583e OpenShift-Rebase-Source: bf7e23e UPSTREAM: <carry>: STOR-829: Add CSIInlineVolumeSecurity admission plugin The CSIInlineVolumeSecurity admission plugin inspects inline CSI volumes on pod creation and compares the security.openshift.io/csi-ephemeral-volume-profile label on the CSIDriver object to the pod security profile on the namespace. OpenShift-Rebase-Source: a65c34b UPSTREAM: <carry>: add icsp,idms,itms validation reject creating icsp with idms/itms exist Reject icsp with idms.itms resources exists. According to the discuusion resolution https://docs.google.com/document/d/13h6IJn8wlzXdiPMvCWlMEHOXXqEZ9_GYOl02Wldb3z8/edit?usp=sharing, one of current icsp or new mirror setting crd should be rejected if a user tries to use them on the same cluster. Signed-off-by: Qi Wang <[email protected]> UPSTREAM: <carry>: node admission plugin for cpu partitioning The ManagedNode admission plugin makes the Infrastructure.Status.CPUPartitioning field authoritative. This validates that nodes that wish to join the cluster are first configured to properly handle workload pinning For more information see - openshift/enhancements#1213 Signed-off-by: ehila <[email protected]> UPSTREAM: <carry>: kube-apiserver: allow injection of kube-apiserver options UPSTREAM: <carry>: kube-apiserver: allow rewiring OpenShift-Rebase-Source: 56b49c9 OpenShift-Rebase-Source: bcf574c UPSTREAM: <carry>: openshift-kube-apiserver: add kube-apiserver patches UPSTREAM: <carry>: openshift-kube-apiserver: add kube-apiserver patches initialize DefaultComponentGlobalsRegistry after feature gates have been parsed from the config

…rver code UPSTREAM: <carry>: openshift-kube-apiserver: enabled conversion gen for admission configs UPSTREAM: <carry>: openshift-kube-apiserver/admission: fix featuregates resource name UPSTREAM: <carry>: openshift-kube-apiserver/admission: add missing FeatureSets UPSTREAM: <carry>: openshift-kube-apiserver: use github.com/openshift/apiserver-library-go/pkg/labelselector UPSTREAM: <carry>: openshift authenticator: don't allow old-style tokens UPSTREAM: <carry>: oauth-authn: support sha256 prefixed tokens UPSTREAM: <carry>: oauth-token-authn: switch to sha256~ prefix UPSTREAM: <carry>: oauth-token-authn: add sha256~ support to bootstrap authenticator UPSTREAM: <drop>: remove the openshift authenticator from the apiserver In 4.8, we moved the authenticator to be configured via webhookTokenAuthenticators to an endpoint in the oauth-apiserver, this should now be safe to remove. UPSTREAM: <carry>: set ResourceQuotaValidationOptions to true When PodAffinityNamespaceSelector goes to beta or GA this might affect how our ClusterResourceQuota might work UPSTREAM: <carry>: simplify the authorizer patch to allow the flags to function UPSTREAM: <carry>: eliminate unnecessary closure in openshift configuration wiring UPSTREAM: <carry>: add crdvalidation for apiserver.spec.tlsSecurityProfile UPSTREAM: <carry>: openshift-kube-apiserver: Add custom resource validation for network spec UPSTREAM: <carry>: stop overriding flags that are explicitly set UPSTREAM: <carry>: add readyz check for openshift apiserver availability UPSTREAM: <carry>: wait for oauth-apiserver accessibility UPSTREAM: <carry>: provide a new admission plugin to mutate management pods CPUs requests The ManagementCPUOverride admission plugin replaces pod container CPU requests with a new management resource. It applies to all pods that: 1. are in an allowed namespace 2. and have the workload annotation. It also sets the new management resource request and limit and set resource annotation that CRI-O can recognize and apply the relevant changes. For more information, see - openshift/enhancements#703 Conditions for CPUs requests deletion: 1. The namespace should have allowed annotation "workload.openshift.io/allowed": "management" 2. The pod should have management annotation: "workload.openshift.io/management": "{"effect": "PreferredDuringScheduling"}" 3. All nodes under the cluster should have new management resource - "management.workload.openshift.io/cores" 4. The CPU request deletion will not change the pod QoS class UPSTREAM: <carry>: Does not prevent pod creation because of no nodes reason when it runs under the regular cluster Check the `cluster` infrastructure resource status to be sure that we run on top of a SNO cluster and in case if the pod runs on top of regular cluster, exit before node existence check. UPSTREAM: <carry>: do not mutate pods when it has a container with both CPU request and limit Removing the CPU request from the container that has a CPU limit will result in the defaulter to set the CPU request back equals to the CPU limit. UPSTREAM: <carry>: Reject the pod creation when we can not decide the cluster type It is possible a race condition between pod creation and the update of the infrastructure resource status with correct values under Status.ControlPlaneTopology and Status.InfrastructureTopology. UPSTREAM: <carry>: add CRD validation for dnses Add an admission plugin that validates the dnses.operator.openshift.io custom resource. For now, the plugin only validates the DNS pod node-placement parameters. This commit fixes bug 1967745. https://bugzilla.redhat.com/show_bug.cgi?id=1967745 * openshift-kube-apiserver/admission/customresourcevalidation/attributes.go (init): Install operatorv1 into supportedObjectsScheme. * openshift-kube-apiserver/admission/customresourcevalidation/customresourcevalidationregistration/cr_validation_registration.go (AllCustomResourceValidators, RegisterCustomResourceValidation): Register the new plugin. * openshift-kube-apiserver/admission/customresourcevalidation/dns/validate_dns.go: New file. (PluginName): New const. (Register): New function. Register the plugin. (toDNSV1): New function. Convert a runtime object to a versioned DNS. (dnsV1): New type to represent a runtime object that is validated as a versioned DNS. (ValidateCreate, ValidateUpdate, ValidateStatusUpdate): New methods. Implement the ObjectValidator interface, using the validateDNSSpecCreate and validateDNSSpecUpdate helpers. (validateDNSSpecCreate, validateDNSSpecUpdate): New functions. Validate a DNS, using the validateDNSSpec helper. (validateDNSSpec): New function. Validate the spec field of a DNS, using the validateDNSNodePlacement helper. (validateDNSNodePlacement): New function. Validate the node selector and tolerations in a DNS's node-placement parameters, using validateTolerations. (validateTolerations): New function. Validate a slice of corev1.Toleration. * openshift-kube-apiserver/admission/customresourcevalidation/dns/validate_dns_test.go: New file. (TestFailValidateDNSSpec): Verify that validateDNSSpec rejects invalid DNS specs. (TestSucceedValidateDNSSpec): Verify that validateDNSSpec accepts valid DNS specs. * vendor/*: Regenerate. UPSTREAM: <carry>: prevent the kubecontrollermanager service-ca from getting less secure UPSTREAM: <carry>: allow SCC to be disabled on a per-namespace basis UPSTREAM: <carry>: verify required http2 cipher suites In the Apiserver admission, we need to return an error if the required http2 cipher suites are missing from a custom tlsSecurityProfile. Currently, custom cipher suites missing ECDHE_RSA_WITH_AES_128_GCM_SHA256 or ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 result in invalid http2 Server configuration causing the apiservers to crash. See: go/x/net/http2.ConfigureServer for futher information. UPSTREAM: <carry>: drop the warning to use --keep-annotations When a user runs the `oc debug` command for the pod with the management resource, we will inform him that he should pass `--keep-annotations` parameter to the debug command. UPSTREAM: <carry>: admission/managementcpusoverride: cover the roll-back case During the upgrade and roll-back flow 4.7->4.8->4.7, the topology related fields under the infrastructure can be empty because the old API does not support them. The code will equal the empty infrastructure section with the current one. When the status has some other non-empty field, and topology fields are empty, we assume that the cluster currently passes via roll-back and not via the clean install. UPSTREAM: <carry>: Remove pod warning annotation when workload partitioning is disabled UPSTREAM: <carry>: use new access token inactivity timeout field. UPSTREAM: <carry>: apirequestcount validation UPSTREAM: <carry>: Added config node object validation for extreme latency profiles UPSTREAM: <carry>: Add Upstream validation in the DNS admission check patches UPSTREAM: <carry>: Make RestrictedEndpointsAdmission check NotReadyAddresses UPSTREAM: <carry>: Make RestrictedEndpointsAdmission restrict EndpointSlices as well Moved SkipSystemMasterAuthorizers to the authorizer. UPSTREAM: <carry>: Add validation plugin for CRD-based route parity. UPSTREAM: <carry>: Add host assignment plugin for CRD-based routes. UPSTREAM: <carry>: Apply shared defaulters to CRD-based routes. Signed-off-by: Artyom Lukianov <[email protected]> Signed-off-by: Damien Grisonnet <[email protected]> Signed-off-by: Swarup Ghosh <[email protected]> OpenShift-Rebase-Source: 932411e OpenShift-Rebase-Source: 1899555 OpenShift-Rebase-Source: 453583e OpenShift-Rebase-Source: bf7e23e UPSTREAM: <carry>: STOR-829: Add CSIInlineVolumeSecurity admission plugin The CSIInlineVolumeSecurity admission plugin inspects inline CSI volumes on pod creation and compares the security.openshift.io/csi-ephemeral-volume-profile label on the CSIDriver object to the pod security profile on the namespace. OpenShift-Rebase-Source: a65c34b UPSTREAM: <carry>: add icsp,idms,itms validation reject creating icsp with idms/itms exist Reject icsp with idms.itms resources exists. According to the discuusion resolution https://docs.google.com/document/d/13h6IJn8wlzXdiPMvCWlMEHOXXqEZ9_GYOl02Wldb3z8/edit?usp=sharing, one of current icsp or new mirror setting crd should be rejected if a user tries to use them on the same cluster. UPSTREAM: <carry>: node admission plugin for cpu partitioning The ManagedNode admission plugin makes the Infrastructure.Status.CPUPartitioning field authoritative. This validates that nodes that wish to join the cluster are first configured to properly handle workload pinning For more information see - openshift/enhancements#1213 UPSTREAM: <carry>: kube-apiserver: allow injection of kube-apiserver options UPSTREAM: <carry>: kube-apiserver: allow rewiring OpenShift-Rebase-Source: 56b49c9 OpenShift-Rebase-Source: bcf574c UPSTREAM: <carry>: STOR-1270: Admission plugin to deny deletion of storages.operator.openshift.io UPSTREAM: <carry>: support for both icsp and idms objects Revert: openshift#1310 Add support for ICSP and IDMS objects living at the same time. UPSTREAM: <carry>: openshift-kube-apiserver: add openshift-kube-apisever code UPSTREAM: <carry>: featureset validation moved to CEL UPSTREAM: <carry>: Add context to ObjectValidator TODO: add router validation logic to implement ctx add in ObjectValidator UPSTREAM: <carry>: loosen authentication.spec.type validation UPSTREAM: <carry>: openshift-kube-apiserver: add kube-apiserver patches pod .spec.nodeName should not override project node selector in podNodeEnvironment admission plugin UPSTREAM: <carry>: Fix sets.String and sets.Set[string] type mismatch libray-go uses the genetic Set while upstream still uses the deprecated sets.String in some part of its codes. UPSTREAM: <carry>: Add RouteExternalCertificate validation in Route ObjectValidator UPSTREAM: <carry>: Fix incorrect type casting in admission validate_apiserver UPSTREAM: <carry>: react to library-go changes UPSTREAM: <carry>: Update RouteExternalCertificate validation in Route ObjectValidator UPSTREAM: <carry>: APIRequestCount Handler OpenShift-Rebase-Source: 4d74b77