[BUG] Attach CNI is very slow in large scale deployment

[jslouisyou]

Describe the bug
Hello, whereabouts team.

I'm facing an issue while I'm trying to deploy large scale deployment (e.g. daemonset or deployment with 600 Pods or above) with SR-IOV VFs. When I deployed, I found that multus and whereabouts takes around 1 minutes to attach CNIs into a Pod, but eventually elapsed some time Pod can't be changed into running state and restarted forever.

Here's the event of Pod:

Events:
  Type     Reason                  Age                    From               Message
  ----     ------                  ----                   ----               -------
  Normal   Scheduled               7m47s                  default-scheduler  Successfully assigned default/mpi-test-9dd76bfb6-pzqc5 to srh100-328
  Normal   AddedInterface          7m46s                  multus             Add eth0 [10.11.173.32/32] from k8s-pod-network
  Normal   AddedInterface          7m34s                  multus             Add net1 [192.168.192.188/20] from default/sriov-gpu-ib0
  Normal   AddedInterface          6m20s                  multus             Add net2 [192.168.209.89/20] from default/sriov-gpu-ib1
  Warning  FailedCreatePodSandBox  3m47s                  kubelet            Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = context deadline exceeded
  Warning  FailedCreatePodSandBox  2m53s (x5 over 3m46s)  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to reserve sandbox name "mpi-test-9dd76bfb6-pzqc5_default_48d74874-65eb-4412-abbd-a9408a5cec28_0": name "mpi-test-9dd76bfb6-pzqc5_default_48d74874-65eb-4412-abbd-a9408a5cec28_0" is reserved for "fed595e15fb89f02f77b05d2be56a0829e5a749944daaf051a5785a537a6d642"
  Normal   SandboxChanged          2m46s                  kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   AddedInterface          100s                   multus             Add eth0 [10.11.173.30/32] from k8s-pod-network
  Normal   AddedInterface          81s                    multus             Add net1 [192.168.192.77/20] from default/sriov-gpu-ib0
  Normal   AddedInterface          15s                    multus             Add net2 [192.168.208.172/20] from default/sriov-gpu-ib1

Here's the log from whereabouts:

2024-10-10T16:53:17+09:00 [debug] OnStartedLeading() called
2024-10-10T16:53:17+09:00 [debug] Elected as leader, do processing
2024-10-10T16:53:17+09:00 [debug] IPManagement -- mode: 0 / containerID: 5fa2f47e0c74b01acc8f9730aa9f16ebbf9e20c79abad089a9086b5c32ec205d / podRef: default/mpi-test-9dd76bfb6-wtvcm
2024-10-10T16:53:17+09:00 [debug] IterateForAssignment input >> range_start: 192.168.240.0 | range_end: <nil> | ipnet: {192.168.240.0 fffff000} | first IP: 192.168.240.1 | last IP: 192.168.255.254
2024-10-10T16:53:17+09:00 [debug] Reserving IP: |192.168.240.141 5fa2f47e0c74b01acc8f9730aa9f16ebbf9e20c79abad089a9086b5c32ec205d|
2024-10-10T16:53:17+09:00 [debug] OnStoppedLeading() called
2024-10-10T16:53:17+09:00 [debug] Finished leader election
2024-10-10T16:53:17+09:00 [debug] IPManagement: [{192.168.240.141 fffff000}], <nil>
2024-10-10T16:53:18+09:00 [debug] Used defaults from parsed flat file config @ /etc/cni/net.d/whereabouts.d/whereabouts.conf
2024-10-10T16:53:18+09:00 [debug] ADD - IPAM configuration successfully read: {Name:sriov_gpu_ib3 Type:whereabouts Routes:[] Addresses:[] IPRanges:[{OmitRanges:[] Range:192.168.240.0/20 RangeStart:192.168.240.0 RangeEnd:<nil>}] OmitRanges:[] DNS:{Nameservers:[] Domain: Search:[] Options:[]} Range: RangeStart:<nil> RangeEnd:<nil> GatewayStr: LeaderLeaseDuration:1500 LeaderRenewDeadline:1000 LeaderRetryPeriod:500 LogFile:/tmp/whereabouts.log LogLevel:debug ReconcilerCronExpression:30 4 * * * OverlappingRanges:false SleepForRace:0 Gateway:<nil> Kubernetes:{KubeConfigPath:/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig K8sAPIRoot:} ConfigurationPath: PodName:mpi-test-9dd76bfb6-sqxh8 PodNamespace:default NetworkName:}
2024-10-10T16:53:18+09:00 [debug] Beginning IPAM for ContainerID: 234ae371527b8eff3f257fce4df48cbf404ebcedf04de9887c2fd956e7781cfe
2024-10-10T16:53:18+09:00 [debug] Started leader election
2024-10-10T16:54:13+09:00 [debug] OnStartedLeading() called
2024-10-10T16:54:13+09:00 [debug] Elected as leader, do processing
2024-10-10T16:54:13+09:00 [debug] IPManagement -- mode: 0 / containerID: 234ae371527b8eff3f257fce4df48cbf404ebcedf04de9887c2fd956e7781cfe / podRef: default/mpi-test-9dd76bfb6-sqxh8
2024-10-10T16:54:13+09:00 [debug] IterateForAssignment input >> range_start: 192.168.240.0 | range_end: <nil> | ipnet: {192.168.240.0 fffff000} | first IP: 192.168.240.1 | last IP: 192.168.255.254
2024-10-10T16:54:13+09:00 [debug] Reserving IP: |192.168.241.9 234ae371527b8eff3f257fce4df48cbf404ebcedf04de9887c2fd956e7781cfe|
2024-10-10T16:54:13+09:00 [debug] OnStoppedLeading() called
2024-10-10T16:54:13+09:00 [debug] Finished leader election
2024-10-10T16:54:13+09:00 [debug] IPManagement: [{192.168.241.9 fffff000}], <nil>
2024-10-10T16:54:14+09:00 [debug] Used defaults from parsed flat file config @ /etc/cni/net.d/whereabouts.d/whereabouts.conf
2024-10-10T16:54:14+09:00 [debug] ADD - IPAM configuration successfully read: {Name:sriov_gpu_ib4 Type:whereabouts Routes:[] Addresses:[] IPRanges:[{OmitRanges:[] Range:192.169.0.0/20 RangeStart:192.169.0.0 RangeEnd:<nil>}] OmitRanges:[] DNS:{Nameservers:[] Domain: Search:[] Options:[]} Range: RangeStart:<nil> RangeEnd:<nil> GatewayStr: LeaderLeaseDuration:1500 LeaderRenewDeadline:1000 LeaderRetryPeriod:500 LogFile:/tmp/whereabouts.log LogLevel:debug ReconcilerCronExpression:30 4 * * * OverlappingRanges:false SleepForRace:0 Gateway:<nil> Kubernetes:{KubeConfigPath:/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig K8sAPIRoot:} ConfigurationPath: PodName:mpi-test-9dd76bfb6-wtvcm PodNamespace:default NetworkName:}
(repeated)

I think whereabouts runs very slow to assign IP address to CNI. Is there anyway to improve performance of whereabouts? It would be very grateful!

Thanks.

Expected behavior
whereabouts can assign IP address to CNI more faster.

To Reproduce
Steps to reproduce the behavior:

1. Deploy daemonset or deployment
2. Watch status of Pods

Environment:

• Whereabouts version : v0.8
• Kubernetes version (use kubectl version): v1.29.5
• Network-attachment-definition (Same context but different resource name - sriov_gpu_###)

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: sriov-gpu-ib2
  namespace: default
spec:
  config: |-
    {
      "cniVersion": "0.3.1",
      "name": "sriov_gpu_ib2",
      "plugins": [
        {
          "type": "ib-sriov",
          "link_state": "enable",
          "rdmaIsolation": true,
          "ibKubernetesEnabled": false,
          "ipam": {
            "datastore": "kubernetes",
            "kubernetes": {
              "kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig"
            },
            "log_file": "/tmp/whereabouts.log",
            "log_level": "debug",
            "type": "whereabouts",
            "enable_overlapping_ranges": false,
            "range": "192.168.224.0/20"
          }
        }
      ]
    }

• Whereabouts configuration (on the host): N/A
• OS (e.g. from /etc/os-release): Ubuntu 22.04
• Kernel (e.g. uname -a): 5.15.0-78-generic
• Others: N/A

Additional info / context
Add any other information / context about the problem here.

[jslouisyou]

Also I can see this logs too:

Events:
  Type     Reason                  Age               From               Message
  ----     ------                  ----              ----               -------
  Normal   Scheduled               3m35s             default-scheduler  Successfully assigned default/mpi-test-xk7f8 to srh100-327
  Normal   AddedInterface          3m32s             multus             Add eth0 [10.0.113.147/32] from k8s-pod-network
  Warning  FailedCreatePodSandBox  91s               kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f5a43eed5deaab07837ea70d06b7ccd420144f048cc14f390afaebd6716827bf": plugin type="multus" name="multus-cni-network" failed (add): [default/mpi-test-xk7f8:sriov_gpu_ib0]: error adding container to network "sriov_gpu_ib0": failed to set up IPAM plugin type "whereabouts" from the device "0000:41:00.6": error at storage engine: time limit exceeded while waiting to become leader
  Normal   SandboxChanged          9s (x8 over 91s)  kubelet            Pod sandbox changed, it will be killed and re-created.

[mlguerrero12]

Hi @jslouisyou, yes, we are aware of this issue. This feature was recently introduced. It is experimental though. Please give it a try.

[jslouisyou]

Thanks @mlguerrero12. But It seems that this feature doesn't include v0.8.0 release version and I'm using whereabouts in production cluster, so I'm not sure whether I can use latest version.
Is there plan for release new version which includes fast IPAM feature?