Use systemd-networkd for aws-ecs-2 and k8s 1.28 variants

[zmrow]

Issue number:
Related to #2449

Description of changes:
Draft while testing is in progress

This PR moves the aws-ecs-2, Kubernetes 1.28, and *-dev variants to systemd-networkd as the network backend.

Testing done:
WIP - will update as testing is completed

• Build each of the variants
• aws-k8s-1.28 conformance testing in IPv6-only cluster

{
"plugins": [
{
"plugin": "e2e",
"node": "global",
"status": "complete",
"result-status": "passed",
"result-counts": {
"passed": 384,
"skipped": 7007
},

• aws-k8s-1.28 conformance testing in IPv4 cluster

 e2e   global   complete   passed   Passed:380, Failed:  0, Remaining:  0

• metal-k8s-1.28 conformance testing

e2e   complete   passed       1   Passed:380, Failed:  0, Remaining:  0

• vmware-k8s-1.28 conformance testing

e2e   global   complete   passed   Passed:380, Failed:  0, Remaining:  0

• aws-ecs-2 internal ECS testing
• Given custom (correct) DNS settings, instance continues to function properly in a cluster and proper interface/resolved configuration is set (set custom settings identical to what comes in via DHCP, observe the settings as Global via resolvectl.)

bash-5.1# cat /etc/systemd/resolved.conf.d/10-resolv.conf 
[Resolve]
DNS=192.168.0.2
Domains=us-west-2.compute.internal

bash-5.1# resolvectl 
Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub
      DNS Servers 192.168.0.2
       DNS Domain us-west-2.compute.internal

Link 2 (eth0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

bash-5.1# cat /etc/systemd/network/10-eth0.network.d/10-dns.conf 
[Network]
DNSDefaultRoute=false
[DHCPv4]
UseDNS=false
UseDomains=false
[DHCPv6]
UseDNS=false
UseDomains=false
[IPv6AcceptRA]
UseDNS=false
UseDomains=false

• AWS - chain the Cilium CNI with AWS VPC CNI
• Scale out an IPv6 and IPv4 cluster to 3000 nodes and ensure they all join properly

$ kubectl get nodes | grep -w Ready | wc -l
3000

• Ensure the system doesn't wait for an interface configured with DHCP and marked as optional via network config. Ensure RequiredForOnline=false

[Match]
Name=eno3
[Link]
RequiredForOnline=false
[Network]
DHCP=ipv4
[DHCPv4]
UseMTU=true

• Ensure the system doesn't wait for a protocol (IPv4/6) marked as optional via network config

[Match]
Name=eno1
[Link]
RequiredForOnline=true
RequiredFamilyForOnline=ipv4
[Network]
DHCP=yes
[DHCPv4]
UseMTU=true
[Ipv6AcceptRA]
UseMTU=true

• Smoke test bonding on metal (See netdog: Add none to bonds for deterministic MACs #3415 and netdog: Add BindCarrier to bond network files #3426 for issues found during bond testing
• Smoke test VLANs on metal

version = 3

[bond0]
kind = "bond"
mode = "active-backup"
interfaces = ["eno1" , "eno2"]
dhcp4 = true

[bond0.monitoring]
miimon-frequency-ms = 100
miimon-updelay-ms = 200
miimon-downdelay-ms = 200

[VLAN43]
kind = "vlan"
device = "bond0"
id = 43
dhcp4 = true

• Test all versions (1-3) of network config work as expected
• Test interfaces configured via MAC address work as expected
• Ensure MTU is set identically (via DHCP) to wicked variants across various platforms

Instance Type	systemd-networkd	wicked
c5.large	9001	9001
m6g.medium	9001	9001
c3.large	9001	9001
c1.xlarge	9001	9001
vmware	1500	1500

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[zmrow]

^ Removes the resolveConf kubelet config change in favor of a separate PR #3395

[zmrow]

^ Rebase onto develop

[zmrow]

^ Flips aws-ecs-2-nvidia and the *-dev variants to systemd-networkd. Good catch @bcressey !

[stmcginnis]

Built aws-k8s-1.28 x86_64 AMIs and deployed to a 1.27 EKS cluster.

Installed cilium 1.14.1 via helm:

helm install cilium cilium/cilium --version 1.14.1 \
  --namespace kube-system \
  --set eni.enabled=true \
  --set ipam.mode=eni \
  --set egressMasqueradeInterfaces=eth0 \
  --set tunnel=disabled \
  --set nodeinit.enabled=false

Verified pods were all happy:

$ kubectl get pods -n kube-system
NAME                               READY   STATUS    RESTARTS   AGE
cilium-4d5gn                       1/1     Running   0          7m36s
cilium-hwxhr                       1/1     Running   0          7m37s
cilium-l7xz6                       1/1     Running   0          7m37s
cilium-operator-66d75c5db6-hldz9   1/1     Running   0          4m27s
cilium-operator-66d75c5db6-vbshr   1/1     Running   0          4m48s
cilium-qz9bh                       1/1     Running   0          7m36s
coredns-647484dc8b-jvnk9           1/1     Running   0          4m7s
coredns-647484dc8b-mwx7r           1/1     Running   0          4m7s
kube-proxy-jgh7f                   1/1     Running   0          7m37s
kube-proxy-m2kjq                   1/1     Running   0          7m37s
kube-proxy-sr9fq                   1/1     Running   0          7m36s
kube-proxy-zlfzc                   1/1     Running   0          7m36s

And checked cilium status:

$ cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:          OK
 \__/¯¯\__/    Operator:        OK
 /¯¯\__/¯¯\    Hubble Relay:    disabled
 \__/¯¯\__/    ClusterMesh:     disabled
    \__/

Deployment        cilium-operator    Desired: 2, Ready: 2/2, Available: 2/2
DaemonSet         cilium             Desired: 4, Ready: 4/4, Available: 4/4
Containers:       cilium             Running: 4
                  cilium-operator    Running: 2
Cluster Pods:     6/6 managed by Cilium
Image versions    cilium             quay.io/cilium/cilium:v1.14.1@sha256:edc1d05ea1365c4a8f6ac6982247d5c145181704894bb698619c3827b6963a72: 4
                  cilium-operator    quay.io/cilium/operator-aws:v1.14.1@sha256:ff57964aefd903456745e53a4697a4f6a026d8fffdb06f53f624a23d23ade37a: 2

Ran connectivity tests:

$ cilium connectivity test

...

✅ All 32 tests (250 actions) successful, 0 tests skipped, 1 scenarios skipped.

Everything looks good!

[heri16]

We are using the BottlerRocket AWS-ECS-2 variant. And it fails to detect other ENIs that has been attached. Anyway to solve this?

[yeazelm]

Hello @heri16, can you cut a new issue for this? The networkd work has been complete for a while and should be attaching ENIs. Cutting a separate issue would let us dive in just on the issue you are facing!