Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] fast ranges not working out of the box #518

Open
dougbtv opened this issue Nov 26, 2024 · 0 comments
Open

[BUG] fast ranges not working out of the box #518

dougbtv opened this issue Nov 26, 2024 · 0 comments

Comments

@dougbtv
Copy link
Member

dougbtv commented Nov 26, 2024

Describe the bug
Using fast ranges results

Expected behavior
The pod should come up, however, it fails with:

  Warning  FailedCreatePodSandBox  7s    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_samplepod_default_514048e8-7b94-4cf7-8711-17adb72f0ac7_0(c88719557a2b8b049c0013ae7c62a98a37841d5961b6ab87e87b2b7c2b578969): error adding pod default_samplepod to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"c88719557a2b8b049c0013ae7c62a98a37841d5961b6ab87e87b2b7c2b578969" Netns:"/var/run/netns/e21fd2c7-625b-41fd-80ec-1d23c18c2ef3" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=samplepod;K8S_POD_INFRA_CONTAINER_ID=c88719557a2b8b049c0013ae7c62a98a37841d5961b6ab87e87b2b7c2b578969;K8S_POD_UID=514048e8-7b94-4cf7-8711-17adb72f0ac7" Path:"" ERRORED: error configuring pod [default/samplepod] networking: [default/samplepod/514048e8-7b94-4cf7-8711-17adb72f0ac7:whereaboutsexample]: error adding container to network "whereaboutsexample": netplugin failed: "2024-11-27T05:09:21+09:00 [debug] Used defaults from parsed flat file config @ /etc/cni/net.d/whereabouts.d/whereabouts.conf\n2024-11-27T05:09:21+09:00 [debug] ADD - IPAM configuration successfully read: {Name:whereaboutsexample Type:whereabouts Routes:[] Addresses:[] IPRanges:[{OmitRanges:[] Range:192.168.2.0/24 RangeStart:192.168.2.0 RangeEnd:<nil>}] OmitRanges:[] DNS:{Nameservers:[] Domain: Search:[] Options:[]} Range: NodeSliceSize:/22 RangeStart:<nil> RangeEnd:<nil> GatewayStr: LeaderLeaseDuration:1500 LeaderRenewDeadline:1000 LeaderRetryPeriod:500 LogFile:/var/log/whereabouts.log LogLevel:debug ReconcilerCronExpression:30 4 * * * OverlappingRanges:true SleepForRace:0 Gateway:<nil> Kubernetes:{KubeConfigPath:/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig K8sAPIRoot:} ConfigurationPath: PodName:samplepod PodNamespace:default NetworkName:donkeykongjr}\n2024-11-27T05:09:21+09:00 [debug] Beginning IPAM for ContainerID: \"c88719557a2b8b049c0013ae7c62a98a37841d5961b6ab87e87b2b7c2b578969\" - podRef: \"default/samplepod\" - ifName: \"net1\"\n2024-11-27T05:09:21+09:00 [debug] discovered current hostname as: labkubedualhost-node-1\n2024-11-27T05:09:21+09:00 [debug] ipam namespace is kube-system\n2024-11-27T05:09:21+09:00 [error] error getting node slice kube-system/donkeykongjr nodeslicepools.whereabouts.cni.cncf.io \"donkeykongjr\" not found\n2024-11-27T05:09:21+09:00 [error] Failed to create leader elector: nodeslicepools.whereabouts.cni.cncf.io \"donkeykongjr\" not found\n2024-11-27T05:09:21+09:00 [debug] Started leader election\nE1127 05:09:21.599030  105556 runtime.go:79] Observed a panic: \"invalid memory address or nil pointer dereference\" (runtime error: invalid memory address or nil pointer dereference)\ngoroutine 41 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x1636d20, 0x2642050})\n\t/go/src/github.com/k8snetworkplumbingwg/whereabouts/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x85\nk8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc00050a380?})\n\t/go/src/github.com/k8snetworkplumbingwg/whereabouts/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x6b\npanic({0x1636d20?, 0x2642050?})\n\t/usr/local/go/src/runtime/panic.go:785 +0x132\nk8s.io/client-go/tools/leaderelection.(*LeaderElector).Run(0x0, {0x1a88a68, 0xc0004fe2d0})\n\t/go/src/github.com/k8snetworkplumbingwg/whereabouts/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:206 +0x43\ngithub.com/k8snetworkplumbingwg/whereabouts/pkg/storage/kubernetes.IPManagement.func2.1()\n\t/go/src/github.com/k8snetworkplumbingwg/whereabouts/pkg/storage/kubernetes/ipam.go:496 +0x68\ncreated by github.com/k8snetworkplumbingwg/whereabouts/pkg/storage/kubernetes.IPManagement.func2 in goroutine 40\n\t/go/src/github.com/k8snetworkplumbingwg/whereabouts/pkg/storage/kubernetes/ipam.go:494 +0x109\npanic: runtime error: invalid memory address or nil pointer dereference [recovered]\n\tpanic: runtime error: invalid memory address or nil pointer dereference\n[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x1495103]\n\ngoroutine 41 [running]:\nk8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc00050a380?})\n\t/go/src/github.com/k8snetworkplumbingwg/whereabouts/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:56 +0xcd\npanic({0x1636d20?, 0x2642050?})\n\t/usr/local/go/src/runtime/panic.go:785 +0x132\nk8s.io/client-go/tools/leaderelection.(*LeaderElector).Run(0x0, {0x1a88a68, 0xc0004fe2d0})\n\t/go/src/github.com/k8snetworkplumbingwg/whereabouts/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:206 +0x43\ngithub.com/k8snetworkplumbingwg/whereabouts/pkg/storage/kubernetes.IPManagement.func2.1()\n\t/go/src/github.com/k8snetworkplumbingwg/whereabouts/pkg/storage/kubernetes/ipam.go:496 +0x68\ncreated by github.com/k8snetworkplumbingwg/whereabouts/pkg/storage/kubernetes.IPManagement.func2 in goroutine 40\n\t/go/src/github.com/k8snetworkplumbingwg/whereabouts/pkg/storage/kubernetes/ipam.go:494 +0x109\n"

This looks like a leader election issue at a glance, with a shorter version being:

Failed to create leader elector: nodeslicepools.whereabouts.cni.cncf.io \"donkeykongjr\" not found

And then panics with:

Observed a panic: \"invalid memory address or nil pointer dereference\"

Which seems to be associated with:

github.com/k8snetworkplumbingwg/whereabouts/pkg/storage/kubernetes.IPManagement.func2.1()\n\t/go/src/github.com/k8snetworkplumbingwg/whereabouts/pkg/storage/kubernetes/ipam.go:496

which is found here @ https://github.com/k8snetworkplumbingwg/whereabouts/blob/master/pkg/storage/kubernetes/ipam.go#L496

To Reproduce

Install whereabouts with:

kubectl apply     -f doc/crds/daemonset-install.yaml     -f doc/crds/whereabouts.cni.cncf.io_ippools.yaml     -f doc/crds/whereabouts.cni.cncf.io_overlappingrangeipreservations.yaml     -f doc/crds/whereabouts.cni.cncf.io_nodeslicepools.yaml     -f doc/crds/node-slice-controller.yaml

Apply this net-attach-def

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: whereabouts-fast-ipam
spec:
  config: '{
    "cniVersion": "0.3.0",
    "name": "whereaboutsexample",
    "type": "macvlan",
    "master": "eth0",
    "mode": "bridge",
    "ipam": {
      "type": "whereabouts",
      "range": "192.168.2.0/24",
      "node_slice_size": "/22",
      "network_name": "donkeykongjr",
      "leader_lease_duration": 1500,
      "leader_renew_deadline": 1000,
      "leader_retry_period": 500,
      "log_file": "/var/log/whereabouts.log",
      "log_level": "debug"
    }
  }'

Create this pod:

apiVersion: v1
kind: Pod
metadata:
  name: samplepod
  annotations:
    k8s.v1.cni.cncf.io/networks: whereabouts-fast-ipam
spec:
  containers:
  - name: samplepod
    command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
    image: alpine

Environment:

  • Whereabouts version : Latest (I also tried one from october 17th which also exhibited the same issue)
  • Kubernetes version (use kubectl version): v1.30.7
  • Network-attachment-definition: (above)
  • Whereabouts configuration (on the host): default
  • OS (e.g. from /etc/os-release): fedora
  • Kernel (e.g. uname -a): 6.8.5-301.fc40.x86_64
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant