diff --git a/keps/prod-readiness/sig-node/34.yaml b/keps/prod-readiness/sig-node/34.yaml new file mode 100644 index 00000000000..721749d15a7 --- /dev/null +++ b/keps/prod-readiness/sig-node/34.yaml @@ -0,0 +1,3 @@ +kep-number: 34 +stable: + approver: "@johnbelamaric" diff --git a/keps/sig-node/34-sysctl-fields/README.md b/keps/sig-node/34-sysctl-fields/README.md index 896668f3867..115de886cfd 100644 --- a/keps/sig-node/34-sysctl-fields/README.md +++ b/keps/sig-node/34-sysctl-fields/README.md @@ -1,23 +1,115 @@ -# Promote sysctl annotations to fields - -## Table of Contents +# KEP-34: Promote sysctl annotations to fields +- [Release Signoff Checklist](#release-signoff-checklist) - [Summary](#summary) - [Motivation](#motivation) - - [Promote annotations to fields](#promote-annotations-to-fields) + - [Goals/Non-Goals](#goalsnon-goals) +- [Proposal](#proposal) + - [Promote annotations to fields (beta)](#promote-annotations-to-fields-beta) - [Promote --experimental-allowed-unsafe-sysctls kubelet flag to kubelet config api option](#promote--kubelet-flag-to-kubelet-config-api-option) - [Gate the feature](#gate-the-feature) -- [Proposal](#proposal) - - [User Stories](#user-stories) - - [Implementation Details/Notes/Constraints](#implementation-detailsnotesconstraints) + - [User Stories (Optional)](#user-stories-optional) + - [Notes/Constraints/Caveats](#notesconstraintscaveats) - [Risks and Mitigations](#risks-and-mitigations) -- [Graduation Criteria](#graduation-criteria) +- [Design Details](#design-details) + - [Test Plan](#test-plan) + - [Graduation Criteria](#graduation-criteria) + - [Alpha](#alpha) + - [Beta](#beta) + - [Graduation](#graduation) + - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) + - [Version Skew Strategy](#version-skew-strategy) +- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) + - [Feature Enablement and Rollback](#feature-enablement-and-rollback) + - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning) + - [Monitoring Requirements](#monitoring-requirements) + - [Dependencies](#dependencies) + - [Scalability](#scalability) + - [Troubleshooting](#troubleshooting) - [Implementation History](#implementation-history) +- [Drawbacks / Alternatives](#drawbacks--alternatives) +- [Infrastructure Needed (Optional)](#infrastructure-needed-optional) +## Release Signoff Checklist + + + +Items marked with (R) are required *prior to targeting to a milestone / release*. + +- [x] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR) +- [x] (R) KEP approvers have approved the KEP status as `implementable` +- [x] (R) Design details are appropriately documented +- [x] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input +- [x] (R) Graduation criteria is in place +- [x] (R) Production readiness review completed +- [x] (R) Production readiness review approved +- [ ] "Implementation History" section is up-to-date for milestone +- [x] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io] +- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes + + + +[kubernetes.io]: https://kubernetes.io/ +[kubernetes/enhancements]: https://git.k8s.io/enhancements +[kubernetes/kubernetes]: https://git.k8s.io/kubernetes +[kubernetes/website]: https://git.k8s.io/website + ## Summary +This proposal aims at extending the current pod specification with support +for namespaced kernel parameters (sysctls) set for each pod. + +See the [abstract and motivation](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/sysctl.md#abstract) from the original proposal in v1.4. + +## Motivation + +See the original design proposal's [motivation section](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/sysctl.md#motivation). + +As mentioned in [contributors/devel/api_changes.md#alpha-field-in-existing-api-version](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md#alpha-field-in-existing-api-version): + +> Previously, annotations were used for experimental alpha features, but are no longer recommended for several reasons: +> +> They expose the cluster to "time-bomb" data added as unstructured annotations against an earlier API server (https://issue.k8s.io/30819) +> They cannot be migrated to first-class fields in the same API version (see the issues with representing a single value in multiple places in backward compatibility gotchas) +> +> The preferred approach adds an alpha field to the existing object, and ensures it is disabled by default: +> +> ... + +The annotations as a means to set `sysctl` are no longer necessary. +The original intent of annotations was to provide additional description of Kubernetes +objects through metadata. +It's time to separate the ability to annotate from the ability to change sysctls settings +so a cluster operator can elevate the distinction between experimental and supported usage +of the feature. + + +### Goals/Non-Goals + +See: original [constraints and assumptions](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/sysctl.md#constraints-and-assumptions) + +## Proposal + +See the [original design proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/sysctl.md#proposed-design) for alpha. + +### Promote annotations to fields (beta) + Setting the `sysctl` parameters through annotations provided a successful story for defining better constraints of running applications. The `sysctl` feature has been tested by a number of people without any serious @@ -61,28 +153,6 @@ in the following way: The `sysctl` design document with more details and rationals is available at [design-proposals/node/sysctl.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/sysctl.md#pod-api-changes) -## Motivation - -As mentioned in [contributors/devel/api_changes.md#alpha-field-in-existing-api-version](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md#alpha-field-in-existing-api-version): - -> Previously, annotations were used for experimental alpha features, but are no longer recommended for several reasons: -> -> They expose the cluster to "time-bomb" data added as unstructured annotations against an earlier API server (https://issue.k8s.io/30819) -> They cannot be migrated to first-class fields in the same API version (see the issues with representing a single value in multiple places in backward compatibility gotchas) -> -> The preferred approach adds an alpha field to the existing object, and ensures it is disabled by default: -> -> ... - -The annotations as a means to set `sysctl` are no longer necessary. -The original intent of annotations was to provide additional description of Kubernetes -objects through metadata. -It's time to separate the ability to annotate from the ability to change sysctls settings -so a cluster operator can elevate the distinction between experimental and supported usage -of the feature. - -### Promote annotations to fields - * Introduce native `sysctl` fields in pods through `spec.securityContext.sysctl` field as: ```yaml @@ -139,11 +209,9 @@ If disabled, the fields and the whitelist are just ignored. [1] https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/ -## Proposal - -This is where we get down to the nitty gritty of what the proposal actually is. +### User Stories (Optional) -### User Stories +See also: [original sysctl proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/sysctl.md#abstract-use-cases) * As a cluster admin, I want to have `sysctl` feature versioned so I can assure backward compatibility and proper transformation between versioned to internal representation and back.. @@ -152,7 +220,7 @@ This is where we get down to the nitty gritty of what the proposal actually is. * As a cluster admin, I want to be able to apply the `sysctl` constraints on the cluster level so I can define the default constraints for all pods. -### Implementation Details/Notes/Constraints +### Notes/Constraints/Caveats Extending `SecurityContext` struct with `Sysctls` field: @@ -188,15 +256,181 @@ Validation checks implemented as part of [#27180](https://github.com/kubernetes/ We need to assure backward compatibility, i.e. object specifications with `sysctl` annotations must still work after the graduation. -## Graduation Criteria +## Design Details + +All of the above details were copied out of earlier proposals. For graduation, the PRR template below is completed. + +### Test Plan + +- Unit tests and e2es for all applicable changes. +- Any required conformance tests for graduation. + +### Graduation Criteria + +#### Alpha + +* add sysctl support to pods +* e2e tests + +Alpha since 1.4. + +#### Beta * API changes allowing to configure the pod-scoped `sysctl` via `spec.securityContext` field. * API changes allowing to configure the cluster-scoped `sysctl` via `PodSecurityPolicy` object -* Promote `--experimental-allowed-unsafe-sysctls` kubelet flag to kubelet config api option * feature gate enabled by default -* e2e tests + +Beta since 1.11. + +#### Graduation + +* Promote `--experimental-allowed-unsafe-sysctls` kubelet flag to kubelet config api option +* lock feature gate on + +### Upgrade / Downgrade Strategy + +There are [e2es](https://github.com/kubernetes/kubernetes/blob/28e2e12b887fe082929d3ece4b3cbd572dc15d39/test/e2e/upgrades/sysctl.go) for sysctl behaviour on upgrades. + +### Version Skew Strategy + +N/A + +## Production Readiness Review Questionnaire + +### Feature Enablement and Rollback + +###### How can this feature be enabled / disabled in a live cluster? + +- [X] Feature gate (also fill in values in `kep.yaml`) + - Feature gate name: Sysctls + - Components depending on the feature gate: kubelet, apiserver +- [ ] Other + - Describe the mechanism: + - Will enabling / disabling the feature require downtime of the control + plane? + - Will enabling / disabling the feature require downtime or reprovisioning + of a node? (Do not assume `Dynamic Kubelet Config` feature is enabled). + +###### Does enabling the feature change any default behavior? + +No. Enabling the feature allows the use of sysctls. + +###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)? + +Yes, disable the feature flag. + +###### What happens if we reenable the feature if it was previously rolled back? + +Feature will become available again on the component. + +###### Are there any tests for feature enablement/disablement? + +Not currently. Feature has defaulted to on since 1.11; graduation criteria would lock feature to on. + +### Rollout, Upgrade and Rollback Planning + +###### How can a rollout fail? Can it impact already running workloads? + +N/A + +###### What specific metrics should inform a rollback? + +N/A + +###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested? + +Yes: https://github.com/kubernetes/kubernetes/blob/28e2e12b887fe082929d3ece4b3cbd572dc15d39/test/e2e/upgrades/sysctl.go + +###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.? + +N/A + +### Monitoring Requirements + +###### How can an operator determine if the feature is in use by workloads? + +No metric currently exists. Feature flag will be set to on and Pod or PSP specifications will include sysctl fields set. + +###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service? + +N/A, not a service. + +###### What are the reasonable SLOs (Service Level Objectives) for the above SLIs? + +N/A, not a service. + +###### Are there any missing metrics that would be useful to have to improve observability of this feature? + +N/A + +### Dependencies + +Underlying kernel support for sysctls. + +###### Does this feature depend on any specific services running in the cluster? + +No. + +### Scalability + +###### Will enabling / using this feature result in any new API calls? + +No. + +###### Will enabling / using this feature result in introducing new API types? + +No. + +###### Will enabling / using this feature result in any new calls to the cloud provider? + +No. + +###### Will enabling / using this feature result in increasing size or count of the existing API objects? + +Yes: pods and PSPs have new fields for sysctl values. + +###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs? + +No. + +###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components? + +No. + +### Troubleshooting + +###### How does this feature react if the API server and/or etcd is unavailable? + +Feature is an API field on pod specification; kubelets behave as usual when API server/etcd are unavailable. + +###### What are other known failure modes? + +- https://github.com/kubernetes/kubernetes/issues/72593 +- https://github.com/kubernetes/kubernetes/issues/74151 + +There may be some follow-ups required to improve usability, but I do not +believe this should block graduation. + +Any scheduling enhancement we make around a node that is configured to allow +unsafe sysctls would be a distinct feature. + +###### What steps should be taken if SLOs are not being met to determine the problem? + +SLOs do not apply, N/A. ## Implementation History -The `sysctl` feature is tracked as part of [features#34](https://github.com/kubernetes/features/issues/34). -This is one of the goals to promote the annotations to fields. +- 2017-06-12: [Original design proposal](https://github.com/kubernetes/community/pull/700) +- 2018-05-14: [Update KEP with beta criteria](https://github.com/kubernetes/community/pull/2093) +- 2018-06-06: [Promote sysctl annotations to fields](https://github.com/kubernetes/kubernetes/pull/63717) +- 2018-06-14: [Update sysctls to beta on website](https://github.com/kubernetes/website/pull/8804) +- 2019-07-02: [Add allowed sysctl to KubeletConfiguration](https://github.com/kubernetes/kubernetes/pull/72974) +- 2021-02-08: Update KEP with final graduation criteria/complete PRR questionnaire + +## Drawbacks / Alternatives + +See also: [original design alternatives and considerations](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/sysctl.md#design-alternatives-and-considerations) + +## Infrastructure Needed (Optional) + +N/A diff --git a/keps/sig-node/34-sysctl-fields/kep.yaml b/keps/sig-node/34-sysctl-fields/kep.yaml index bbe6a32fb55..1d5e3917cc6 100644 --- a/keps/sig-node/34-sysctl-fields/kep.yaml +++ b/keps/sig-node/34-sysctl-fields/kep.yaml @@ -4,17 +4,30 @@ authors: - "@ingvagabund" owning-sig: sig-node participating-sigs: + - sig-node - sig-auth reviewers: - "@sjenning" - "@derekwaynecarr" approvers: - - "@sjenning " - "@derekwaynecarr" +prr-approvers: + - "@johnbelamaric" editor: creation-date: 2018-04-30 -last-updated: 2018-05-02 -status: provisional -see-also: -replaces: -superseded-by: +last-updated: 2021-02-08 +status: implementable + +# The target maturity stage in the current dev cycle for this KEP. +stage: stable + +# The most recent milestone for which work toward delivery of this KEP has been +# done. This can be the current (upcoming) milestone, if it is being actively +# worked on. +latest-milestone: "v1.21" + +# The milestone at which this feature was, or is targeted to be, at each stage. +milestone: + alpha: "v1.4" + beta: "v1.11" + stable: "v1.21"