Container control plane

[aerosouund]

What this PR does / why we need it:

Runs the control plane of the kubevirtCI cluster in containers to discount on resource consumption, expects to reduce a runner per CI lane run across all repos
It is based on #1230

In a typical kubevirtCI cluster, the control plane is unschedulable. As seen in this snippet, only the system containers are on it

[vagrant@node01 ~]$ sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf get pods -A -o wide | grep 101
kube-system   calico-node-xwvhg                          1/1     Running   0          3m47s   192.168.66.101   node01   <none>           <none>
kube-system   etcd-node01                                1/1     Running   1          4m1s    192.168.66.101   node01   <none>           <none>
kube-system   kube-apiserver-node01                      1/1     Running   1          4m1s    192.168.66.101   node01   <none>           <none>
kube-system   kube-controller-manager-node01             1/1     Running   1          4m1s    192.168.66.101   node01   <none>           <none>
kube-system   kube-proxy-qcwmn                           1/1     Running   0          3m47s   192.168.66.101   node01   <none>           <none>
kube-system   kube-scheduler-node01                      1/1     Running   1          4m2s    192.168.66.101   node01   <none>           <none>

Design

There is no standardized tool or technology that achieves what this PR tries to achieve. Atleast not in a way that matches the needs of kubevirtCI. One of the big requirements is that the control plane joining process remains the same for workers (through kubeadm) whether you are joining a VM control plane or a container control plane and so the PR provides its own way of provisioning certificates, running DNS in the cluster and many other rudimentary k8s concepts.

The code for the control plane container lives in cluster-provision/gocli/control-plane.
The creation of the control plane happens in a way similar to how kubeadm provisions a cluster, through the concept of phases. below is a snippet from its main function to illustrate how some of the phases are being called

	if err := NewCertsPhase(defaultPkiPath).Run(); err != nil {
		return nil, err
	}

	if err := NewRunETCDPhase(cp.dnsmasqID, cp.containerRuntime, defaultPkiPath).Run(); err != nil {
		return nil, err
	}

	if err := NewKubeConfigPhase(defaultPkiPath).Run(); err != nil {
		return nil, err
	}

	if err := NewRunControlPlaneComponentsPhase(cp.dnsmasqID, cp.containerRuntime, defaultPkiPath, cp.k8sVersion).Run(); err != nil {
		return nil, err
	}

The phases it runs are:

• Certs: provisioning the cluster certificate authority and the certificates of individual components signed by this CA
• ETCD: Runs etcd
• Kubeconfig: Create the admin, controller manager & scheduler kubeconfig files
• Bootstrappers RBAC: Create a bootstrap token secret and the necessary RBAC roles for the kubelet to register itself to the api
• Bootstrap auth resources: Create important resources that kubeadm expects to find in the cluster during joining
• Kube Proxy: Deploy Kube Proxy
• CNI: Create the Calico CNI that would be previously created by node01
• CoreDNS: Deploy CoreDNS

This then gets instantiated in the KV provider to start it

	if kp.Nodes > 1 {
		runner := controlplane.NewControlPlaneRunner(dnsmasq, strings.Split(kp.Version, "-")[1], uint(kp.APIServerPort))
		c, err = runner.Start()
		if err != nil {
			return err
		}
		k8sClient, err := k8s.NewDynamicClient(c)
		if err != nil {
			return err
		}
		kp.Client = k8sClient
	}

and node01 will now only get called if the node count is 1

	if nodeIdx == 1 && kp.Nodes == 1 {
		n := node01.NewNode01Provisioner(sshClient, kp.SingleStack, kp.NoEtcdFsync)

Changes to the networking setup

Only one change is required in dnsmasq.sh

  if [ ${NUM_NODES} -gt 1 ] && [ $i -eq 1 ]; then
    ip tuntap add dev tap101 mode tap user $(whoami)
    ip link set tap101 master br0
    ip link set dev tap101 up
    ip addr add 192.168.66.110/24 dev tap101
    ip -6 addr add fd00::110 dev tap101
    iptables -t nat -A PREROUTING -p tcp -i eth0 -m tcp --dport 6443 -j DNAT --to-destination 192.168.66.110:6443
  fi

If the node count is higher than 1 create an interface called tap101 and manually assign it the required IPs and forward port 6443 on it through eth0. Since the api server container gets launched in the same netns as dnsmasq all whats needed afterwards is that the server advertises the IP 192.168.66.110 as the api server endpoint. and this is taken care of in the code

Current state

The PR is under testing locally to see if all previously existing functionality is retained. As well as code cleaning for the final presentation. The PR is opened as a draft for the community to see it and give feedback on any points to help drive the direction of the project due to its size

Extra notes
cc: @dhiller @brianmcarey @acardace @xpivarc

Checklist

This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

• Design: A design document was considered and is present (link) or not required
• PR: The PR description is expressive enough and will help future contributors
• Code: Write code that humans can understand and Keep it simple
• Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
• Upgrade: Impact of this change on upgrade flows was considered and addressed if required
• Testing: New code requires new unit tests. New features and bug fixes require at least on e2e test
• Documentation: A user-guide update was considered and is present (link) or not required. You want a user-guide update if it's a user facing feature / API change.
• Community: Announcement to kubevirt-dev was considered

Release note:

[kubevirt-bot]

Thanks for your pull request. Before we can look at it, you'll need to add a 'DCO signoff' to your commits.

📝 Please follow instructions in the contributing guide to update your commits with the DCO

Full details of the Developer Certificate of Origin can be found at developercertificate.org.

The list of commits missing DCO signoff:

• 0bd7601 control plane stuff
• c22baab control plane stuff
• 71edf3b use net parse ip

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[brianmcarey]

Hi @aerosouund - thanks for all the effort on this.

I have a couple of concerns about this.

Overall, I am not convinced of the resources that this will save as we will just be running the kubernetes control plane components somewhere else.

We will still have a requirement to have a control plane node for our testing in kubevirt/kubevirt as a lot of the KubeVirt infra components require a control plane node to be scheduled - kubevirt/kubevirt#11659

These KubeVirt infra components can be sensitive to things like selinux policies and kernel modules so running in a controlled VM adds some benefits here.

We could make this container control plane configurable but I don't see it running in the main CI workloads and I am not sure how much it will be used in local dev environments as if a someone wants to test against a lightweight cluster setup, we have the kind cluster providers which provides a very lightweight cluster.

Let me know what you think. These are just a couple of the issues that came to my mind on this.

[aerosouund]

@brianmcarey

Valid concerns for sure, let me discuss them

Overall, I am not convinced of the resources that this will save as we will just be running the kubernetes control plane components somewhere else.

True that the control plane will run somewhere else, but you would not be reserving a big amount of compute for it. Currently, whatever amount you reserve for VMs in the cluster is what gets allocated to the control plane which can be rather big sometimes.

In general, the majority of resource saving is accrued from being given the ability to schedule workload pods (istio, cdi, multus.. etc) on any node in the cluster. Meaning that you have unlocked the total sum of resources used by those containers to be allocatable on node 1 which was previously the control plane.

We will still have a requirement to have a control plane node for our testing in kubevirt/kubevirt as a lot of the KubeVirt infra components require a control plane node to be scheduled - kubevirt/kubevirt#11659

Yes, i ran into this issue while testing this PR. and for now the current hacky fix i am using is by labeling a random node as the control plane even though it isn't and that seems to be enough to make it work.

But if we check the reasons mentioned in the PR, they say its because

if you take over virt-controller, you can create a pod in every namspace with an image of your choosing, mounting any secret you like and dumping it.
if you take over virt-operator you can directly create privileged pods
if you take over virt-api you can inject via modifying webhooks your own malicius kernel, image

I am still reading their full rationale behind this, but based on what they say these risks will be present wherever the components are scheduled.
And also to my knowledge (and correct me if i am wrong), we aren't providing any additional security hardening on the control plane node. So the problems are present in all cases.

And an opinionated take i have on this is that KubeVirtCI is an ephemeral cluster creator and not meant for any long lived clusters. Also, clusters it creates run on isolated environments (atleast in CI thats how it is) . And so in terms of priorities CI efficiency and resources beats security

These KubeVirt infra components can be sensitive to things like selinux policies and kernel modules so running in a controlled VM adds some benefits here.

If i understand correctly you are saying KV infra components are best suited for being ran on a VM. Well, under this PR they still are.
What has changed is that the control plane components are running elsewhere and all nodes are being labeled as workers (which frees up node01)

We could make this container control plane configurable but I don't see it running in the main CI workloads and I am not sure how much it will be used in local dev environments as if a someone wants to test against a lightweight cluster setup, we have the kind cluster providers which provides a very lightweight cluster.

It has challenges for sure, but i believe that it has very strong candidacy to run in CI and no challenge (so far) seems so glaring to imply its impossible to run it in CI. The latest of them being the timing out of validation webhooks due to the api server not being in the pod network. This was overcame by using konnectivity.

Happy to discuss this further. Let me know if you have any opinions on what i said or if anything i said needs correction

[brianmcarey]

True that the control plane will run somewhere else, but you would not be reserving a big amount of compute for it. Currently, whatever amount you reserve for VMs in the cluster is what gets allocated to the control plane which can be rather big sometimes.

In general, the majority of resource saving is accrued from being given the ability to schedule workload pods (istio, cdi, multus.. etc) on any node in the cluster. Meaning that you have unlocked the total sum of resources used by those containers to be allocatable on node 1 which was previously the control plane.

We do use the control plane node (node01) for scheduling test workloads so I am not sure if the resources are wasted. node01 is schedulable. For instance here you can see a test VM on node01 - https://storage.googleapis.com/kubevirt-prow/pr-logs/pull/kubevirt_kubevirt/13208/pull-kubevirt-e2e-k8s-1.31-sig-compute/1859000686096683008/artifacts/k8s-reporter/3/1_overview.log

What would the benefit of this approach be over just running a single node kubevirtci cluster? What kind of resource savings are seeing by moving the kubeernetes control components out of the VM? I don't think does components are that heavy on resources but maybe I am wrong.

[aerosouund]

@brianmcarey

We do use the control plane node (node01) for scheduling test workloads so I am not sure if the resources are wasted. node01 is schedulable. For instance here you can see a test VM on node01 - https://storage.googleapis.com/kubevirt-prow/pr-logs/pull/kubevirt_kubevirt/13208/pull-kubevirt-e2e-k8s-1.31-sig-compute/1859000686096683008/artifacts/k8s-reporter/3/1_overview.log

Based on this it seems that some components are indeed scheduled on the control plane, I need to investigate per component why that is as some components actively check for the control plane label on the node to be scheduled on them (kubevirt CR job for example), but by default the control plane is not taking any pods on it.
My take is at tests with sufficient scale those components (KubeVirtCI components) can indeed take up sizable resources. I can try to provide hard numbers for this

What would the benefit of this approach be over just running a single node kubevirtci cluster? What kind of resource savings are seeing by moving the kubeernetes control components out of the VM? I don't think does components are that heavy on resources but maybe I am wrong.

You are right in saying that moving the control plane components isn't gonna result in high savings as they aren't the main culprit in resource consumption. The benefit is that by taking them out, all your nodes are now workers and you can treat them all as a shared resource pool, rather than giving a particular node special treatment. This means that the kubevirtci components don't need to take into account scheduling laws or things like that and can sit on any node. Effectively achieving better resource utilization and savings across the cluster.

In general, the sensible way for move forward with this project is to:

1- Get it working for all KubeVirtCI test cases and lanes
2- See if we can actually start downsizing some testing workloads (by changing the job definition) and if we see improvements we can have this be our new standard

Let me know what you think