From 2e96733e4b71ac97065be56a97a30ed548bb0f5e Mon Sep 17 00:00:00 2001
From: Elana Hashman <ehashman@redhat.com>
Date: Mon, 8 Feb 2021 14:41:22 -0800
Subject: [PATCH 1/2] Migrate KEP-34 to new template

---
 keps/prod-readiness/sig-node/34.yaml     |   3 +
 keps/sig-node/34-sysctl-fields/README.md | 463 ++++++++++++++++++++++-
 keps/sig-node/34-sysctl-fields/kep.yaml  |  25 +-
 3 files changed, 471 insertions(+), 20 deletions(-)
 create mode 100644 keps/prod-readiness/sig-node/34.yaml

diff --git a/keps/prod-readiness/sig-node/34.yaml b/keps/prod-readiness/sig-node/34.yaml
new file mode 100644
index 00000000000..721749d15a7
--- /dev/null
+++ b/keps/prod-readiness/sig-node/34.yaml
@@ -0,0 +1,3 @@
+kep-number: 34
+stable:
+  approver: "@johnbelamaric"
diff --git a/keps/sig-node/34-sysctl-fields/README.md b/keps/sig-node/34-sysctl-fields/README.md
index 896668f3867..87f25b325e3 100644
--- a/keps/sig-node/34-sysctl-fields/README.md
+++ b/keps/sig-node/34-sysctl-fields/README.md
@@ -1,21 +1,77 @@
-# Promote sysctl annotations to fields
-
-## Table of Contents
+# KEP-34: Promote sysctl annotations to fields
 
 <!-- toc -->
+- [Release Signoff Checklist](#release-signoff-checklist)
 - [Summary](#summary)
 - [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
   - [Promote annotations to fields](#promote-annotations-to-fields)
   - [Promote <code>--experimental-allowed-unsafe-sysctls</code> kubelet flag to kubelet config api option](#promote--kubelet-flag-to-kubelet-config-api-option)
   - [Gate the feature](#gate-the-feature)
-- [Proposal](#proposal)
-  - [User Stories](#user-stories)
-  - [Implementation Details/Notes/Constraints](#implementation-detailsnotesconstraints)
+  - [User Stories (Optional)](#user-stories-optional)
+  - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional)
   - [Risks and Mitigations](#risks-and-mitigations)
-- [Graduation Criteria](#graduation-criteria)
+- [Design Details](#design-details)
+  - [Test Plan](#test-plan)
+  - [Graduation Criteria](#graduation-criteria)
+    - [Alpha](#alpha)
+    - [Beta](#beta)
+    - [Graduation](#graduation)
+  - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy)
+  - [Version Skew Strategy](#version-skew-strategy)
+- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire)
+  - [Feature Enablement and Rollback](#feature-enablement-and-rollback)
+  - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning)
+  - [Monitoring Requirements](#monitoring-requirements)
+  - [Dependencies](#dependencies)
+  - [Scalability](#scalability)
+  - [Troubleshooting](#troubleshooting)
 - [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+- [Infrastructure Needed (Optional)](#infrastructure-needed-optional)
 <!-- /toc -->
 
+## Release Signoff Checklist
+
+<!--
+**ACTION REQUIRED:** In order to merge code into a release, there must be an
+issue in [kubernetes/enhancements] referencing this KEP and targeting a release
+milestone **before the [Enhancement Freeze](https://git.k8s.io/sig-release/releases)
+of the targeted release**.
+
+For enhancements that make changes to code or processes/procedures in core
+Kubernetes—i.e., [kubernetes/kubernetes], we require the following Release
+Signoff checklist to be completed.
+
+Check these off as they are completed for the Release Team to track. These
+checklist items _must_ be updated for the enhancement to be released.
+-->
+
+Items marked with (R) are required *prior to targeting to a milestone / release*.
+
+- [x] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR)
+- [x] (R) KEP approvers have approved the KEP status as `implementable`
+- [x] (R) Design details are appropriately documented
+- [x] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input
+- [x] (R) Graduation criteria is in place
+- [x] (R) Production readiness review completed
+- [x] (R) Production readiness review approved
+- [ ] "Implementation History" section is up-to-date for milestone
+- [x] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
+- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
+
+<!--
+**Note:** This checklist is iterative and should be reviewed and updated every time this enhancement is being considered for a milestone.
+-->
+
+[kubernetes.io]: https://kubernetes.io/
+[kubernetes/enhancements]: https://git.k8s.io/enhancements
+[kubernetes/kubernetes]: https://git.k8s.io/kubernetes
+[kubernetes/website]: https://git.k8s.io/website
+
 ## Summary
 
 Setting the `sysctl` parameters through annotations provided a successful story
@@ -81,6 +137,22 @@ It's time to separate the ability to annotate from the ability to change sysctls
 so a cluster operator can elevate the distinction between experimental and supported usage
 of the feature.
 
+### Goals
+
+<!--
+List the specific goals of the KEP. What is it trying to achieve? How will we
+know that this has succeeded?
+-->
+
+### Non-Goals
+
+<!--
+What is out of scope for this KEP? Listing non-goals helps to focus discussion
+and make progress.
+-->
+
+## Proposal
+
 ### Promote annotations to fields
 
 * Introduce native `sysctl` fields in pods through `spec.securityContext.sysctl` field as:
@@ -139,11 +211,7 @@ If disabled, the fields and the whitelist are just ignored.
 
 [1] https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/
 
-## Proposal
-
-This is where we get down to the nitty gritty of what the proposal actually is.
-
-### User Stories
+### User Stories (Optional)
 
 * As a cluster admin, I want to have `sysctl` feature versioned so I can assure backward compatibility
   and proper transformation between versioned to internal representation and back..
@@ -152,7 +220,7 @@ This is where we get down to the nitty gritty of what the proposal actually is.
 * As a cluster admin, I want to be able to apply the `sysctl` constraints on the cluster level so
   I can define the default constraints for all pods.
 
-### Implementation Details/Notes/Constraints
+### Notes/Constraints/Caveats (Optional)
 
 Extending `SecurityContext` struct with `Sysctls` field:
 
@@ -188,7 +256,42 @@ Validation checks implemented as part of [#27180](https://github.com/kubernetes/
 We need to assure backward compatibility, i.e. object specifications with `sysctl` annotations
 must still work after the graduation.
 
-## Graduation Criteria
+## Design Details
+
+<!--
+This section should contain enough information that the specifics of your
+change are understandable. This may include API specs (though not always
+required) or even code snippets. If there's any ambiguity about HOW your
+proposal will be implemented, this is the place to discuss them.
+-->
+
+### Test Plan
+
+<!--
+**Note:** *Not required until targeted at a release.*
+
+Consider the following in developing a test plan for this enhancement:
+- Will there be e2e and integration tests, in addition to unit tests?
+- How will it be tested in isolation vs with other components?
+
+No need to outline all of the test cases, just the general strategy. Anything
+that would count as tricky in the implementation, and anything particularly
+challenging to test, should be called out.
+
+All code is expected to have adequate tests (eventually with coverage
+expectations). Please adhere to the [Kubernetes testing guidelines][testing-guidelines]
+when drafting this test plan.
+
+[testing-guidelines]: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
+-->
+
+### Graduation Criteria
+
+#### Alpha
+
+#### Beta
+
+#### Graduation
 
 * API changes allowing to configure the pod-scoped `sysctl` via `spec.securityContext` field.
 * API changes allowing to configure the cluster-scoped `sysctl` via `PodSecurityPolicy` object
@@ -196,7 +299,339 @@ must still work after the graduation.
 * feature gate enabled by default
 * e2e tests
 
+### Upgrade / Downgrade Strategy
+
+<!--
+If applicable, how will the component be upgraded and downgraded? Make sure
+this is in the test plan.
+
+Consider the following in developing an upgrade/downgrade strategy for this
+enhancement:
+- What changes (in invocations, configurations, API use, etc.) is an existing
+  cluster required to make on upgrade, in order to maintain previous behavior?
+- What changes (in invocations, configurations, API use, etc.) is an existing
+  cluster required to make on upgrade, in order to make use of the enhancement?
+-->
+
+### Version Skew Strategy
+
+<!--
+If applicable, how will the component handle version skew with other
+components? What are the guarantees? Make sure this is in the test plan.
+
+Consider the following in developing a version skew strategy for this
+enhancement:
+- Does this enhancement involve coordinating behavior in the control plane and
+  in the kubelet? How does an n-2 kubelet without this feature available behave
+  when this feature is used?
+- Will any other components on the node change? For example, changes to CSI,
+  CRI or CNI may require updating that component before the kubelet.
+-->
+
+## Production Readiness Review Questionnaire
+
+<!--
+
+Production readiness reviews are intended to ensure that features merging into
+Kubernetes are observable, scalable and supportable; can be safely operated in
+production environments, and can be disabled or rolled back in the event they
+cause increased failures in production. See more in the PRR KEP at
+https://git.k8s.io/enhancements/keps/sig-architecture/1194-prod-readiness.
+
+The production readiness review questionnaire must be completed and approved
+for the KEP to move to `implementable` status and be included in the release.
+
+In some cases, the questions below should also have answers in `kep.yaml`. This
+is to enable automation to verify the presence of the review, and to reduce review
+burden and latency.
+
+The KEP must have a approver from the
+[`prod-readiness-approvers`](http://git.k8s.io/enhancements/OWNERS_ALIASES)
+team. Please reach out on the
+[#prod-readiness](https://kubernetes.slack.com/archives/CPNHUMN74) channel if
+you need any help or guidance.
+-->
+
+### Feature Enablement and Rollback
+
+<!--
+This section must be completed when targeting alpha to a release.
+-->
+
+###### How can this feature be enabled / disabled in a live cluster?
+
+<!--
+Pick one of these and delete the rest.
+-->
+
+- [ ] Feature gate (also fill in values in `kep.yaml`)
+  - Feature gate name:
+  - Components depending on the feature gate:
+- [ ] Other
+  - Describe the mechanism:
+  - Will enabling / disabling the feature require downtime of the control
+    plane?
+  - Will enabling / disabling the feature require downtime or reprovisioning
+    of a node? (Do not assume `Dynamic Kubelet Config` feature is enabled).
+
+###### Does enabling the feature change any default behavior?
+
+<!--
+Any change of default behavior may be surprising to users or break existing
+automations, so be extremely careful here.
+-->
+
+###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?
+
+<!--
+Describe the consequences on existing workloads (e.g., if this is a runtime
+feature, can it break the existing applications?).
+
+NOTE: Also set `disable-supported` to `true` or `false` in `kep.yaml`.
+-->
+
+###### What happens if we reenable the feature if it was previously rolled back?
+
+###### Are there any tests for feature enablement/disablement?
+
+<!--
+The e2e framework does not currently support enabling or disabling feature
+gates. However, unit tests in each component dealing with managing data, created
+with and without the feature, are necessary. At the very least, think about
+conversion tests if API types are being modified.
+-->
+
+### Rollout, Upgrade and Rollback Planning
+
+<!--
+This section must be completed when targeting beta to a release.
+-->
+
+###### How can a rollout fail? Can it impact already running workloads?
+
+<!--
+Try to be as paranoid as possible - e.g., what if some components will restart
+mid-rollout?
+-->
+
+###### What specific metrics should inform a rollback?
+
+<!--
+What signals should users be paying attention to when the feature is young
+that might indicate a serious problem?
+-->
+
+###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?
+
+<!--
+Describe manual testing that was done and the outcomes.
+Longer term, we may want to require automated upgrade/rollback tests, but we
+are missing a bunch of machinery and tooling and can't do that now.
+-->
+
+###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?
+
+<!--
+Even if applying deprecation policies, they may still surprise some users.
+-->
+
+### Monitoring Requirements
+
+<!--
+This section must be completed when targeting beta to a release.
+-->
+
+###### How can an operator determine if the feature is in use by workloads?
+
+<!--
+Ideally, this should be a metric. Operations against the Kubernetes API (e.g.,
+checking if there are objects with field X set) may be a last resort. Avoid
+logs or events for this purpose.
+-->
+
+###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
+
+<!--
+Pick one more of these and delete the rest.
+-->
+
+- [ ] Metrics
+  - Metric name:
+  - [Optional] Aggregation method:
+  - Components exposing the metric:
+- [ ] Other (treat as last resort)
+  - Details:
+
+###### What are the reasonable SLOs (Service Level Objectives) for the above SLIs?
+
+<!--
+At a high level, this usually will be in the form of "high percentile of SLI
+per day <= X". It's impossible to provide comprehensive guidance, but at the very
+high level (needs more precise definitions) those may be things like:
+  - per-day percentage of API calls finishing with 5XX errors <= 1%
+  - 99% percentile over day of absolute value from (job creation time minus expected
+    job creation time) for cron job <= 10%
+  - 99,9% of /health requests per day finish with 200 code
+-->
+
+###### Are there any missing metrics that would be useful to have to improve observability of this feature?
+
+<!--
+Describe the metrics themselves and the reasons why they weren't added (e.g., cost,
+implementation difficulties, etc.).
+-->
+
+### Dependencies
+
+<!--
+This section must be completed when targeting beta to a release.
+-->
+
+###### Does this feature depend on any specific services running in the cluster?
+
+<!--
+Think about both cluster-level services (e.g. metrics-server) as well
+as node-level agents (e.g. specific version of CRI). Focus on external or
+optional services that are needed. For example, if this feature depends on
+a cloud provider API, or upon an external software-defined storage or network
+control plane.
+
+For each of these, fill in the following—thinking about running existing user workloads
+and creating new ones, as well as about cluster-level services (e.g. DNS):
+  - [Dependency name]
+    - Usage description:
+      - Impact of its outage on the feature:
+      - Impact of its degraded performance or high-error rates on the feature:
+-->
+
+### Scalability
+
+<!--
+For alpha, this section is encouraged: reviewers should consider these questions
+and attempt to answer them.
+
+For beta, this section is required: reviewers must answer these questions.
+
+For GA, this section is required: approvers should be able to confirm the
+previous answers based on experience in the field.
+-->
+
+###### Will enabling / using this feature result in any new API calls?
+
+<!--
+Describe them, providing:
+  - API call type (e.g. PATCH pods)
+  - estimated throughput
+  - originating component(s) (e.g. Kubelet, Feature-X-controller)
+Focusing mostly on:
+  - components listing and/or watching resources they didn't before
+  - API calls that may be triggered by changes of some Kubernetes resources
+    (e.g. update of object X triggers new updates of object Y)
+  - periodic API calls to reconcile state (e.g. periodic fetching state,
+    heartbeats, leader election, etc.)
+-->
+
+###### Will enabling / using this feature result in introducing new API types?
+
+<!--
+Describe them, providing:
+  - API type
+  - Supported number of objects per cluster
+  - Supported number of objects per namespace (for namespace-scoped objects)
+-->
+
+###### Will enabling / using this feature result in any new calls to the cloud provider?
+
+<!--
+Describe them, providing:
+  - Which API(s):
+  - Estimated increase:
+-->
+
+###### Will enabling / using this feature result in increasing size or count of the existing API objects?
+
+<!--
+Describe them, providing:
+  - API type(s):
+  - Estimated increase in size: (e.g., new annotation of size 32B)
+  - Estimated amount of new objects: (e.g., new Object X for every existing Pod)
+-->
+
+###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?
+
+<!--
+Look at the [existing SLIs/SLOs].
+
+Think about adding additional work or introducing new steps in between
+(e.g. need to do X to start a container), etc. Please describe the details.
+
+[existing SLIs/SLOs]: https://git.k8s.io/community/sig-scalability/slos/slos.md#kubernetes-slisslos
+-->
+
+###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components?
+
+<!--
+Things to keep in mind include: additional in-memory state, additional
+non-trivial computations, excessive access to disks (including increased log
+volume), significant amount of data sent and/or received over network, etc.
+This through this both in small and large cases, again with respect to the
+[supported limits].
+
+[supported limits]: https://git.k8s.io/community//sig-scalability/configs-and-limits/thresholds.md
+-->
+
+### Troubleshooting
+
+<!--
+This section must be completed when targeting beta to a release.
+
+The Troubleshooting section currently serves the `Playbook` role. We may consider
+splitting it into a dedicated `Playbook` document (potentially with some monitoring
+details). For now, we leave it here.
+-->
+
+###### How does this feature react if the API server and/or etcd is unavailable?
+
+###### What are other known failure modes?
+
+<!--
+For each of them, fill in the following information by copying the below template:
+  - [Failure mode brief description]
+    - Detection: How can it be detected via metrics? Stated another way:
+      how can an operator troubleshoot without logging into a master or worker node?
+    - Mitigations: What can be done to stop the bleeding, especially for already
+      running user workloads?
+    - Diagnostics: What are the useful log messages and their required logging
+      levels that could help debug the issue?
+      Not required until feature graduated to beta.
+    - Testing: Are there any tests for failure mode? If not, describe why.
+-->
+
+###### What steps should be taken if SLOs are not being met to determine the problem?
+
 ## Implementation History
 
 The `sysctl` feature is tracked as part of [features#34](https://github.com/kubernetes/features/issues/34).
 This is one of the goals to promote the annotations to fields.
+
+## Drawbacks
+
+<!--
+Why should this KEP _not_ be implemented?
+-->
+
+## Alternatives
+
+<!--
+What other approaches did you consider, and why did you rule them out? These do
+not need to be as detailed as the proposal, but should include enough
+information to express the idea and why it was not acceptable.
+-->
+
+## Infrastructure Needed (Optional)
+
+<!--
+Use this section if you need things from the project/SIG. Examples include a
+new subproject, repos requested, or GitHub details. Listing these here allows a
+SIG to get the process for these resources started right away.
+-->
diff --git a/keps/sig-node/34-sysctl-fields/kep.yaml b/keps/sig-node/34-sysctl-fields/kep.yaml
index bbe6a32fb55..1d5e3917cc6 100644
--- a/keps/sig-node/34-sysctl-fields/kep.yaml
+++ b/keps/sig-node/34-sysctl-fields/kep.yaml
@@ -4,17 +4,30 @@ authors:
   - "@ingvagabund"
 owning-sig: sig-node
 participating-sigs:
+  - sig-node
   - sig-auth
 reviewers:
   - "@sjenning"
   - "@derekwaynecarr"
 approvers:
-  - "@sjenning "
   - "@derekwaynecarr"
+prr-approvers:
+  - "@johnbelamaric"
 editor:
 creation-date: 2018-04-30
-last-updated: 2018-05-02
-status: provisional
-see-also:
-replaces:
-superseded-by:
+last-updated: 2021-02-08
+status: implementable
+
+# The target maturity stage in the current dev cycle for this KEP.
+stage: stable
+
+# The most recent milestone for which work toward delivery of this KEP has been
+# done. This can be the current (upcoming) milestone, if it is being actively
+# worked on.
+latest-milestone: "v1.21"
+
+# The milestone at which this feature was, or is targeted to be, at each stage.
+milestone:
+  alpha: "v1.4"
+  beta: "v1.11"
+  stable: "v1.21"

From 61bc65c3422cedba4fb09da71b8f2c3cb605e7ee Mon Sep 17 00:00:00 2001
From: Elana Hashman <ehashman@redhat.com>
Date: Mon, 8 Feb 2021 17:33:28 -0800
Subject: [PATCH 2/2] Fill out PRR, graduation criteria, and implementation
 history

---
 keps/sig-node/34-sysctl-fields/README.md | 413 ++++++-----------------
 1 file changed, 106 insertions(+), 307 deletions(-)

diff --git a/keps/sig-node/34-sysctl-fields/README.md b/keps/sig-node/34-sysctl-fields/README.md
index 87f25b325e3..115de886cfd 100644
--- a/keps/sig-node/34-sysctl-fields/README.md
+++ b/keps/sig-node/34-sysctl-fields/README.md
@@ -4,14 +4,13 @@
 - [Release Signoff Checklist](#release-signoff-checklist)
 - [Summary](#summary)
 - [Motivation](#motivation)
-  - [Goals](#goals)
-  - [Non-Goals](#non-goals)
+  - [Goals/Non-Goals](#goalsnon-goals)
 - [Proposal](#proposal)
-  - [Promote annotations to fields](#promote-annotations-to-fields)
+  - [Promote annotations to fields (beta)](#promote-annotations-to-fields-beta)
   - [Promote <code>--experimental-allowed-unsafe-sysctls</code> kubelet flag to kubelet config api option](#promote--kubelet-flag-to-kubelet-config-api-option)
   - [Gate the feature](#gate-the-feature)
   - [User Stories (Optional)](#user-stories-optional)
-  - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
   - [Risks and Mitigations](#risks-and-mitigations)
 - [Design Details](#design-details)
   - [Test Plan](#test-plan)
@@ -29,8 +28,7 @@
   - [Scalability](#scalability)
   - [Troubleshooting](#troubleshooting)
 - [Implementation History](#implementation-history)
-- [Drawbacks](#drawbacks)
-- [Alternatives](#alternatives)
+- [Drawbacks / Alternatives](#drawbacks--alternatives)
 - [Infrastructure Needed (Optional)](#infrastructure-needed-optional)
 <!-- /toc -->
 
@@ -74,6 +72,44 @@ Items marked with (R) are required *prior to targeting to a milestone / release*
 
 ## Summary
 
+This proposal aims at extending the current pod specification with support
+for namespaced kernel parameters (sysctls) set for each pod.
+
+See the [abstract and motivation](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/sysctl.md#abstract) from the original proposal in v1.4.
+
+## Motivation
+
+See the original design proposal's [motivation section](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/sysctl.md#motivation).
+
+As mentioned in [contributors/devel/api_changes.md#alpha-field-in-existing-api-version](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md#alpha-field-in-existing-api-version):
+
+> Previously, annotations were used for experimental alpha features, but are no longer recommended for several reasons:
+>
+>    They expose the cluster to "time-bomb" data added as unstructured annotations against an earlier API server (https://issue.k8s.io/30819)
+>    They cannot be migrated to first-class fields in the same API version (see the issues with representing a single value in multiple places in backward compatibility gotchas)
+>
+> The preferred approach adds an alpha field to the existing object, and ensures it is disabled by default:
+>
+> ...
+
+The annotations as a means to set `sysctl` are no longer necessary.
+The original intent of annotations was to provide additional description of Kubernetes
+objects through metadata.
+It's time to separate the ability to annotate from the ability to change sysctls settings
+so a cluster operator can elevate the distinction between experimental and supported usage
+of the feature.
+
+
+### Goals/Non-Goals
+
+See: original [constraints and assumptions](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/sysctl.md#constraints-and-assumptions)
+
+## Proposal
+
+See the [original design proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/sysctl.md#proposed-design) for alpha.
+
+### Promote annotations to fields (beta)
+
 Setting the `sysctl` parameters through annotations provided a successful story
 for defining better constraints of running applications.
 The `sysctl` feature has been tested by a number of people without any serious
@@ -117,44 +153,6 @@ in the following way:
 
 The `sysctl` design document with more details and rationals is available at [design-proposals/node/sysctl.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/sysctl.md#pod-api-changes)
 
-## Motivation
-
-As mentioned in [contributors/devel/api_changes.md#alpha-field-in-existing-api-version](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md#alpha-field-in-existing-api-version):
-
-> Previously, annotations were used for experimental alpha features, but are no longer recommended for several reasons:
->
->    They expose the cluster to "time-bomb" data added as unstructured annotations against an earlier API server (https://issue.k8s.io/30819)
->    They cannot be migrated to first-class fields in the same API version (see the issues with representing a single value in multiple places in backward compatibility gotchas)
->
-> The preferred approach adds an alpha field to the existing object, and ensures it is disabled by default:
->
-> ...
-
-The annotations as a means to set `sysctl` are no longer necessary.
-The original intent of annotations was to provide additional description of Kubernetes
-objects through metadata.
-It's time to separate the ability to annotate from the ability to change sysctls settings
-so a cluster operator can elevate the distinction between experimental and supported usage
-of the feature.
-
-### Goals
-
-<!--
-List the specific goals of the KEP. What is it trying to achieve? How will we
-know that this has succeeded?
--->
-
-### Non-Goals
-
-<!--
-What is out of scope for this KEP? Listing non-goals helps to focus discussion
-and make progress.
--->
-
-## Proposal
-
-### Promote annotations to fields
-
 * Introduce native `sysctl` fields in pods through `spec.securityContext.sysctl` field as:
 
   ```yaml
@@ -213,6 +211,8 @@ If disabled, the fields and the whitelist are just ignored.
 
 ### User Stories (Optional)
 
+See also: [original sysctl proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/sysctl.md#abstract-use-cases)
+
 * As a cluster admin, I want to have `sysctl` feature versioned so I can assure backward compatibility
   and proper transformation between versioned to internal representation and back..
 * As a cluster admin, I want to be confident the `sysctl` feature is stable enough and well supported so
@@ -220,7 +220,7 @@ If disabled, the fields and the whitelist are just ignored.
 * As a cluster admin, I want to be able to apply the `sysctl` constraints on the cluster level so
   I can define the default constraints for all pods.
 
-### Notes/Constraints/Caveats (Optional)
+### Notes/Constraints/Caveats
 
 Extending `SecurityContext` struct with `Sysctls` field:
 
@@ -258,115 +258,52 @@ must still work after the graduation.
 
 ## Design Details
 
-<!--
-This section should contain enough information that the specifics of your
-change are understandable. This may include API specs (though not always
-required) or even code snippets. If there's any ambiguity about HOW your
-proposal will be implemented, this is the place to discuss them.
--->
+All of the above details were copied out of earlier proposals. For graduation, the PRR template below is completed.
 
 ### Test Plan
 
-<!--
-**Note:** *Not required until targeted at a release.*
+- Unit tests and e2es for all applicable changes.
+- Any required conformance tests for graduation.
 
-Consider the following in developing a test plan for this enhancement:
-- Will there be e2e and integration tests, in addition to unit tests?
-- How will it be tested in isolation vs with other components?
+### Graduation Criteria
 
-No need to outline all of the test cases, just the general strategy. Anything
-that would count as tricky in the implementation, and anything particularly
-challenging to test, should be called out.
+#### Alpha
 
-All code is expected to have adequate tests (eventually with coverage
-expectations). Please adhere to the [Kubernetes testing guidelines][testing-guidelines]
-when drafting this test plan.
+* add sysctl support to pods
+* e2e tests
 
-[testing-guidelines]: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
--->
+Alpha since 1.4.
 
-### Graduation Criteria
+#### Beta
 
-#### Alpha
+* API changes allowing to configure the pod-scoped `sysctl` via `spec.securityContext` field.
+* API changes allowing to configure the cluster-scoped `sysctl` via `PodSecurityPolicy` object
+* feature gate enabled by default
 
-#### Beta
+Beta since 1.11.
 
 #### Graduation
 
-* API changes allowing to configure the pod-scoped `sysctl` via `spec.securityContext` field.
-* API changes allowing to configure the cluster-scoped `sysctl` via `PodSecurityPolicy` object
 * Promote `--experimental-allowed-unsafe-sysctls` kubelet flag to kubelet config api option
-* feature gate enabled by default
-* e2e tests
+* lock feature gate on
 
 ### Upgrade / Downgrade Strategy
 
-<!--
-If applicable, how will the component be upgraded and downgraded? Make sure
-this is in the test plan.
-
-Consider the following in developing an upgrade/downgrade strategy for this
-enhancement:
-- What changes (in invocations, configurations, API use, etc.) is an existing
-  cluster required to make on upgrade, in order to maintain previous behavior?
-- What changes (in invocations, configurations, API use, etc.) is an existing
-  cluster required to make on upgrade, in order to make use of the enhancement?
--->
+There are [e2es](https://github.com/kubernetes/kubernetes/blob/28e2e12b887fe082929d3ece4b3cbd572dc15d39/test/e2e/upgrades/sysctl.go) for sysctl behaviour on upgrades.
 
 ### Version Skew Strategy
 
-<!--
-If applicable, how will the component handle version skew with other
-components? What are the guarantees? Make sure this is in the test plan.
-
-Consider the following in developing a version skew strategy for this
-enhancement:
-- Does this enhancement involve coordinating behavior in the control plane and
-  in the kubelet? How does an n-2 kubelet without this feature available behave
-  when this feature is used?
-- Will any other components on the node change? For example, changes to CSI,
-  CRI or CNI may require updating that component before the kubelet.
--->
+N/A
 
 ## Production Readiness Review Questionnaire
 
-<!--
-
-Production readiness reviews are intended to ensure that features merging into
-Kubernetes are observable, scalable and supportable; can be safely operated in
-production environments, and can be disabled or rolled back in the event they
-cause increased failures in production. See more in the PRR KEP at
-https://git.k8s.io/enhancements/keps/sig-architecture/1194-prod-readiness.
-
-The production readiness review questionnaire must be completed and approved
-for the KEP to move to `implementable` status and be included in the release.
-
-In some cases, the questions below should also have answers in `kep.yaml`. This
-is to enable automation to verify the presence of the review, and to reduce review
-burden and latency.
-
-The KEP must have a approver from the
-[`prod-readiness-approvers`](http://git.k8s.io/enhancements/OWNERS_ALIASES)
-team. Please reach out on the
-[#prod-readiness](https://kubernetes.slack.com/archives/CPNHUMN74) channel if
-you need any help or guidance.
--->
-
 ### Feature Enablement and Rollback
 
-<!--
-This section must be completed when targeting alpha to a release.
--->
-
 ###### How can this feature be enabled / disabled in a live cluster?
 
-<!--
-Pick one of these and delete the rest.
--->
-
-- [ ] Feature gate (also fill in values in `kep.yaml`)
-  - Feature gate name:
-  - Components depending on the feature gate:
+- [X] Feature gate (also fill in values in `kep.yaml`)
+  - Feature gate name: Sysctls
+  - Components depending on the feature gate: kubelet, apiserver
 - [ ] Other
   - Describe the mechanism:
   - Will enabling / disabling the feature require downtime of the control
@@ -376,262 +313,124 @@ Pick one of these and delete the rest.
 
 ###### Does enabling the feature change any default behavior?
 
-<!--
-Any change of default behavior may be surprising to users or break existing
-automations, so be extremely careful here.
--->
+No. Enabling the feature allows the use of sysctls.
 
 ###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?
 
-<!--
-Describe the consequences on existing workloads (e.g., if this is a runtime
-feature, can it break the existing applications?).
-
-NOTE: Also set `disable-supported` to `true` or `false` in `kep.yaml`.
--->
+Yes, disable the feature flag.
 
 ###### What happens if we reenable the feature if it was previously rolled back?
 
+Feature will become available again on the component.
+
 ###### Are there any tests for feature enablement/disablement?
 
-<!--
-The e2e framework does not currently support enabling or disabling feature
-gates. However, unit tests in each component dealing with managing data, created
-with and without the feature, are necessary. At the very least, think about
-conversion tests if API types are being modified.
--->
+Not currently. Feature has defaulted to on since 1.11; graduation criteria would lock feature to on.
 
 ### Rollout, Upgrade and Rollback Planning
 
-<!--
-This section must be completed when targeting beta to a release.
--->
-
 ###### How can a rollout fail? Can it impact already running workloads?
 
-<!--
-Try to be as paranoid as possible - e.g., what if some components will restart
-mid-rollout?
--->
+N/A
 
 ###### What specific metrics should inform a rollback?
 
-<!--
-What signals should users be paying attention to when the feature is young
-that might indicate a serious problem?
--->
+N/A
 
 ###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?
 
-<!--
-Describe manual testing that was done and the outcomes.
-Longer term, we may want to require automated upgrade/rollback tests, but we
-are missing a bunch of machinery and tooling and can't do that now.
--->
+Yes: https://github.com/kubernetes/kubernetes/blob/28e2e12b887fe082929d3ece4b3cbd572dc15d39/test/e2e/upgrades/sysctl.go
 
 ###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?
 
-<!--
-Even if applying deprecation policies, they may still surprise some users.
--->
+N/A
 
 ### Monitoring Requirements
 
-<!--
-This section must be completed when targeting beta to a release.
--->
-
 ###### How can an operator determine if the feature is in use by workloads?
 
-<!--
-Ideally, this should be a metric. Operations against the Kubernetes API (e.g.,
-checking if there are objects with field X set) may be a last resort. Avoid
-logs or events for this purpose.
--->
+No metric currently exists. Feature flag will be set to on and Pod or PSP specifications will include sysctl fields set.
 
 ###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
 
-<!--
-Pick one more of these and delete the rest.
--->
-
-- [ ] Metrics
-  - Metric name:
-  - [Optional] Aggregation method:
-  - Components exposing the metric:
-- [ ] Other (treat as last resort)
-  - Details:
+N/A, not a service.
 
 ###### What are the reasonable SLOs (Service Level Objectives) for the above SLIs?
 
-<!--
-At a high level, this usually will be in the form of "high percentile of SLI
-per day <= X". It's impossible to provide comprehensive guidance, but at the very
-high level (needs more precise definitions) those may be things like:
-  - per-day percentage of API calls finishing with 5XX errors <= 1%
-  - 99% percentile over day of absolute value from (job creation time minus expected
-    job creation time) for cron job <= 10%
-  - 99,9% of /health requests per day finish with 200 code
--->
+N/A, not a service.
 
 ###### Are there any missing metrics that would be useful to have to improve observability of this feature?
 
-<!--
-Describe the metrics themselves and the reasons why they weren't added (e.g., cost,
-implementation difficulties, etc.).
--->
+N/A
 
 ### Dependencies
 
-<!--
-This section must be completed when targeting beta to a release.
--->
+Underlying kernel support for sysctls.
 
 ###### Does this feature depend on any specific services running in the cluster?
 
-<!--
-Think about both cluster-level services (e.g. metrics-server) as well
-as node-level agents (e.g. specific version of CRI). Focus on external or
-optional services that are needed. For example, if this feature depends on
-a cloud provider API, or upon an external software-defined storage or network
-control plane.
-
-For each of these, fill in the following—thinking about running existing user workloads
-and creating new ones, as well as about cluster-level services (e.g. DNS):
-  - [Dependency name]
-    - Usage description:
-      - Impact of its outage on the feature:
-      - Impact of its degraded performance or high-error rates on the feature:
--->
+No.
 
 ### Scalability
 
-<!--
-For alpha, this section is encouraged: reviewers should consider these questions
-and attempt to answer them.
-
-For beta, this section is required: reviewers must answer these questions.
-
-For GA, this section is required: approvers should be able to confirm the
-previous answers based on experience in the field.
--->
-
 ###### Will enabling / using this feature result in any new API calls?
 
-<!--
-Describe them, providing:
-  - API call type (e.g. PATCH pods)
-  - estimated throughput
-  - originating component(s) (e.g. Kubelet, Feature-X-controller)
-Focusing mostly on:
-  - components listing and/or watching resources they didn't before
-  - API calls that may be triggered by changes of some Kubernetes resources
-    (e.g. update of object X triggers new updates of object Y)
-  - periodic API calls to reconcile state (e.g. periodic fetching state,
-    heartbeats, leader election, etc.)
--->
+No.
 
 ###### Will enabling / using this feature result in introducing new API types?
 
-<!--
-Describe them, providing:
-  - API type
-  - Supported number of objects per cluster
-  - Supported number of objects per namespace (for namespace-scoped objects)
--->
+No.
 
 ###### Will enabling / using this feature result in any new calls to the cloud provider?
 
-<!--
-Describe them, providing:
-  - Which API(s):
-  - Estimated increase:
--->
+No.
 
 ###### Will enabling / using this feature result in increasing size or count of the existing API objects?
 
-<!--
-Describe them, providing:
-  - API type(s):
-  - Estimated increase in size: (e.g., new annotation of size 32B)
-  - Estimated amount of new objects: (e.g., new Object X for every existing Pod)
--->
+Yes: pods and PSPs have new fields for sysctl values.
 
 ###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?
 
-<!--
-Look at the [existing SLIs/SLOs].
-
-Think about adding additional work or introducing new steps in between
-(e.g. need to do X to start a container), etc. Please describe the details.
-
-[existing SLIs/SLOs]: https://git.k8s.io/community/sig-scalability/slos/slos.md#kubernetes-slisslos
--->
+No.
 
 ###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components?
 
-<!--
-Things to keep in mind include: additional in-memory state, additional
-non-trivial computations, excessive access to disks (including increased log
-volume), significant amount of data sent and/or received over network, etc.
-This through this both in small and large cases, again with respect to the
-[supported limits].
-
-[supported limits]: https://git.k8s.io/community//sig-scalability/configs-and-limits/thresholds.md
--->
+No.
 
 ### Troubleshooting
 
-<!--
-This section must be completed when targeting beta to a release.
-
-The Troubleshooting section currently serves the `Playbook` role. We may consider
-splitting it into a dedicated `Playbook` document (potentially with some monitoring
-details). For now, we leave it here.
--->
-
 ###### How does this feature react if the API server and/or etcd is unavailable?
 
+Feature is an API field on pod specification; kubelets behave as usual when API server/etcd are unavailable.
+
 ###### What are other known failure modes?
 
-<!--
-For each of them, fill in the following information by copying the below template:
-  - [Failure mode brief description]
-    - Detection: How can it be detected via metrics? Stated another way:
-      how can an operator troubleshoot without logging into a master or worker node?
-    - Mitigations: What can be done to stop the bleeding, especially for already
-      running user workloads?
-    - Diagnostics: What are the useful log messages and their required logging
-      levels that could help debug the issue?
-      Not required until feature graduated to beta.
-    - Testing: Are there any tests for failure mode? If not, describe why.
--->
+- https://github.com/kubernetes/kubernetes/issues/72593 
+- https://github.com/kubernetes/kubernetes/issues/74151
 
-###### What steps should be taken if SLOs are not being met to determine the problem?
+There may be some follow-ups required to improve usability, but I do not
+believe this should block graduation.
 
-## Implementation History
+Any scheduling enhancement we make around a node that is configured to allow
+unsafe sysctls would be a distinct feature.
 
-The `sysctl` feature is tracked as part of [features#34](https://github.com/kubernetes/features/issues/34).
-This is one of the goals to promote the annotations to fields.
+###### What steps should be taken if SLOs are not being met to determine the problem?
 
-## Drawbacks
+SLOs do not apply, N/A.
 
-<!--
-Why should this KEP _not_ be implemented?
--->
+## Implementation History
 
-## Alternatives
+- 2017-06-12: [Original design proposal](https://github.com/kubernetes/community/pull/700)
+- 2018-05-14: [Update KEP with beta criteria](https://github.com/kubernetes/community/pull/2093)
+- 2018-06-06: [Promote sysctl annotations to fields](https://github.com/kubernetes/kubernetes/pull/63717)
+- 2018-06-14: [Update sysctls to beta on website](https://github.com/kubernetes/website/pull/8804)
+- 2019-07-02: [Add allowed sysctl to KubeletConfiguration](https://github.com/kubernetes/kubernetes/pull/72974)
+- 2021-02-08: Update KEP with final graduation criteria/complete PRR questionnaire
 
-<!--
-What other approaches did you consider, and why did you rule them out? These do
-not need to be as detailed as the proposal, but should include enough
-information to express the idea and why it was not acceptable.
--->
+## Drawbacks / Alternatives
+
+See also: [original design alternatives and considerations](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/sysctl.md#design-alternatives-and-considerations)
 
 ## Infrastructure Needed (Optional)
 
-<!--
-Use this section if you need things from the project/SIG. Examples include a
-new subproject, repos requested, or GitHub details. Listing these here allows a
-SIG to get the process for these resources started right away.
--->
+N/A