cluster/gce: Add env var to enable apiserver basic audit log.

[enisoc]

For now, this is focused on a fixed set of flags that makes the audit
log show up under /var/log/kube-apiserver-audit.log and behave similarly
to /var/log/kube-apiserver.log. Allowing other customization would
require significantly more complex changes.

Audit log rotation is handled the same as for kube-apiserver.log.

What this PR does / why we need it:

Add a knob to enable basic audit logging in GCE.

Which issue this PR fixes:

Special notes for your reviewer:

We would like to cherrypick/port this to release-1.5 also.

Release note:

The kube-apiserver [basic audit log](https://kubernetes.io/docs/admin/audit/) can be enabled in GCE by exporting the environment variable `ENABLE_APISERVER_BASIC_AUDIT=true` before running `cluster/kube-up.sh`. This will log to `/var/log/kube-apiserver-audit.log` and use the same `logrotate` settings as `/var/log/kube-apiserver.log`.

[k8s-reviewable]

This change is 

[cjcullen]

It looks like these logs are about 300B/request, so 1MiB/s translates into ~3500 Req/s. That seems reasonable to me, but could we be spammy enough to bring this into the O(days) category? I think it would fill up in 1 day at ~40M req/s, which is almost certainly higher than any one machine can handle, right?

[enisoc]

40M QPS does seem higher than you'd reasonably expect. At 1.3M QPS, you could hit the limit in 30 days. Even that seems rather high for a kube-apiserver though. There is room in the 31 bits to double this limit if you want. I only chose 1000000000 so it's obvious to humans that it's an arbitrary large number. 2000000000 would also have mostly the same effect.

The long-term fix is to add a new flag or special value for --audit-log-maxsize that makes it bypass Lumberjack and write directly to a file. However, I was hoping to avoid touching anything in the actual binary to make this change minimally invasive. Maybe I can do that as a separate PR on master only after we get this one in and cherrypicked into 1.5?

[cjcullen]

SGTM

[zmerlynn]

Is there a reason you don't configure logrotate for this log? That would solve this comment, I think.

I'm actually pretty concerned with anything log-like that can grow unbounded.

[enisoc]

The idea here is to disable the audit log feature's built-in log rotation (Lumberjack) so we can use the same logrotate facility we already have for kube-apiserver.log.

The current logrotate settings rotate everything matching the wildcard /var/log/*.log so this new /var/log/kube-apiserver-audit.log is included. I verified that it rotates as expected.

https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/gci/configure-helper.sh#L124

[enisoc]

Oh wait I just realized the container-linux version doesn't share the logrotate config. In that case, I think I should revert my change from container-linux and only update GCI since that's what I am able to test.

[zmerlynn]

Ah, GCI/COS is covered. I was looking at the Salt config, which looks like it needs to be updated? https://github.com/kubernetes/kubernetes/blob/master/cluster/saltbase/salt/logrotate/init.sls

[enisoc]

Ah so that's where logrotate is configured for non-GCI. I've pushed an update to that file. Thanks!

[zmerlynn]

This more or less LGTM, except the comment above.

[eparis]

I feel like most of these things are using 'defined' to mean true instead of needing to have the exact magic value/string "true". but I'm ok with this if you really want it this way.

[eparis]

is it wrong that I like the params += style you used the other 2 times. And is sometimes used in this file?

[enisoc]

I like += better too, but I wanted to follow the style used within each file so it's self-consistent.

I'm actually thinking about reverting my change to the trusty and container-linux versions because I don't know how log rotation works on those and updating those other flavors is beyond the scope of what I'm trying to accomplish (getting this working for GCE w/ GCI).

[eparis]

/approve

[enisoc]

I feel like most of these things are using 'defined' to mean true instead of needing to have the exact magic value/string "true". but I'm ok with this if you really want it this way.

I preferred to require "true" because other env vars like ENABLE_GARBAGE_COLLECTOR have set the precedent that you can set it to "false" to disable it. That works in those cases because the value is passed to a boolean command-line flag if non-empty. In this case, there is no boolean command-line flag to enable/disable audit logging, so I would have to explicitly check for "false" to avoid enabling the log.

Given the choice between checking for "true" and checking for "false", I chose "true" because I saw many examples of doing that: ENABLE_MANIFEST_URL, ENABLE_CLUSTER_AUTOSCALER, etc.

[zmerlynn]

Yes, please check for true. I don't know why there are other styles in here at all, but e.g. GKE tends to just pass FOO={true,false} in the kube-env.

[k8s-github-robot]

[APPROVALNOTIFIER] This PR is APPROVED

The following people have approved this PR: enisoc, eparis

Needs approval from an approver in each of these OWNERS Files:

• cluster/OWNERS [eparis]
• cluster/gce/container-linux/OWNERS [eparis]
• hack/OWNERS [eparis]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[zmerlynn]

/lgtm

[k8s-ci-robot]

@zmerlynn: you can't LGTM a PR unless you are an assignee.

In response to this comment:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[zmerlynn]

/lgtm

[zmerlynn]

@k8s-bot cvm gke e2e test this

[k8s-github-robot]

Automatic merge from submit-queue (batch tested with PRs 40297, 41285, 41211, 41243, 39735)

[k8s-cherrypick-bot]

Commit found in the "release-1.5" branch appears to be this PR. Removing the "cherrypick-candidate" label. If this is an error find help to get your PR picked.