Adopt CIS Kubernetes Benchmark, Part 1: API Server.

[JunSun17]

What this PR does / why we need it: Multiple issues are reported by running Kubernetes Auto Analyzer (https://github.com/nccgroup/kube-auto-analyzer). This PR tries to address those issues that can be easily fixed.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #1828

Special notes for your reviewer: No need to review yet, will go through test first to see if it breaks things.

Release note:

[anhowe]

how does this affect each existing kubernetes version including those created by acs-engine, and those created by services that use acs-engine?

[JunSun17]

These changes are based on CIS benchmark, for example, "--anonymous-auth=false" is recommended here:
https://workbench.cisecurity.org/sections/35552/recommendations/98632

I think ideally, each version of Kubernetes should have their own security guidelines, which translates into a bunch of recommended argument settings in yaml files.

For now, the CIS benchmark I used is for Kubernetes 1.8, the recommendations looks pretty generic across multiple versions of Kubernetes to me. So I have not see the needs to do some further versioning yet. Plus, these changes will need to pass multiple version e2e tests before they will hit prod.

Yes, it should affect both acs-engine and the RPs that consumes it.

[jackfrancis]

lgtm

[karataliu]

This breaks acs-engine generated cluster for existing versions.

For the following pod, it would succeed with acs-engine v1.10 generated k8s v1.8 cluster, but fail on acs-engine v1.12 generated k8s v1.8 cluster. Due to SecurityContextDeny option introduced.

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    a: ss
spec:
  containers:
  - name: nginx
    image: nginx
    securityContext:
      seLinuxOptions:
        level: s0
    ports:
    - containerPort: 80

Suppose we should support configurable 'admission-control': provide a recommended value as default, and allow user to override.

---

Update: Looks like APIServerConfig is already configurable. Just need to ensure not to break existing user experience.

[jackfrancis]

@karataliu Thanks for reporting, could you give an example workflow that will fail without the manual config override? (And can you clarify which config override is now necessary for older clusters?)

[JunSun17]

@karataliu Thank you for reporting the issue! Actually the "admission-control" parameter has been made user configurable after this PR. I will investigate the issue you reported to see if the "SecurityContextDeny" should be removed from the default config.

[karataliu]

@jackfrancis @JunSun17
To be clear, there are 2 things here:

• Since default config changes, using default config in v0.10.0 and v0.12.0 to create v1.8 cluster will result out in cluster with different behaviour
• If user wants features blocked by SecurityContextDeny/DenyEscalatingExec, manually overriding is required for v1.8 cluster with v0.12 acs-engine,

Test yaml:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    a: ss
spec:
  containers:
  - name: nginx
    image: nginx
    securityContext:
      seLinuxOptions:
        level: s0
    ports:
    - containerPort: 80

Three combinations:

1. v0.10.0 acs-engine
   apimodel:

"orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorRelease": "1.8",
      "kubernetesConfig": {}
    },

Result: Pod run successfully
pod "nginx" created

2. v0.12.0 acs-engine
   apimodel:

"orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorRelease": "1.8",
      "kubernetesConfig": {}
    },

Result: Failure

Error from server (Forbidden): error when creating "ng.yaml": pods "nginx" is forbidden: SecurityContext.SELinuxOptions is forbidden

3. v0.12.0 acs-engine with config override
   apimodel:

"orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorRelease": "1.8",
      "kubernetesConfig": {
        "apiServerConfig": {
          "--admission-control":  "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota"
        }
      }
    },

Result: Pod run successfully
pod "nginx" created

[JunSun17]

@karataliu Thanks for providing the details on this issue.

I checked the Kubernetes document on this section: https://kubernetes.io/docs/admin/admission-controllers/#securitycontextdeny, and it says

SecurityContextDeny
This admission controller will deny any pod that attempts to set certain escalating SecurityContext fields. This should be enabled if a cluster doesn’t utilize pod security policies to restrict the set of values a security context can take.

Because we have not implement Pod Security Policy, my understanding is the above statement asks to set the SecurityContextDeny flag in admission control.

I do not know how many contains are actively setting their own pod level security context, and thus are impacted by this default setting. If it is a common case, I think it makes sense to remove this flag for now and probably implement Pod Security Policy to mediate this issue. Feel free to share your opinion on this.

[jackfrancis]

@JunSun17 Pod Security Policy as an option went in yesterday in fact:

#2048

Take a look at that and see if there's a path forward.

[JunSun17]

@jackfrancis @karataliu Great, #2048 just come in in time. I see the seLinux policy is set as:

+  seLinux:
+    # This policy assumes the nodes are using AppArmor rather than SELinux.
+    rule: 'RunAsAny'

@karataliu do we need to set nginx security context? Is it come in by default or by customization?

[karataliu]

Since acs-engine by default uses ubuntu as host vm image, selinux is thus unavailable in this case.

I found those when running k8s e2e tests, some of which would fail on a v0.12.0 acs-engine generated cluster.

The selinux configuration comes with following:
https://github.com/kubernetes/kubernetes/blob/c1cd70ad160395160ed64b78ffe8aafdab042352/test/e2e/common/empty_dir.go#L450

Other failures are also related to the admission-control parameter change:

Message: "pods \"pod-34652c7c-fc27-11e7-b1ff-0242ac110003\" is forbidden: pod.Spec.SecurityContext.SELinuxOptions is forbidden",
Message: "pods \"privileged-pod\" is forbidden: cannot exec into or attach to a privileged container",
Message: "pods \"pod-configmaps-bd1a308f-fc28-11e7-bbc0-0242ac110003\" is forbidden: pod.Spec.SecurityContext.RunAsUser is forbidden",

As we can see, SecurityContextDeny blocks pod.Spec.SecurityContext.SELinuxOptions and pod.Spec.SecurityContext.RunAsUser, and DenyEscalatingExec blocks privileged-pod.

[JunSun17]

@karataliu I did some testing and verified by making some changes in PR #2048, using a proper PSP file, the nginx deployment works. It should be the long term solution and documented in Issue #2123.

For now, to reduce impact of this setting, I will remove SecurityContextDeny, as in PR #2125.