kubeadm - can't override detected ip address using config file

[Congelli501]

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version:
kubeadm version: &version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:36:44Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version: 1.13.1
• Cloud provider or hardware configuration: Test VMs (KVM)
• OS: Ubuntu 18.04
• Kernel: Linux master1 4.15.0-43-generic kubeadm should have custom flags for net.IP slice #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Others: 2 NICs on the VM:
  • the first with a default route
  • a second on a private network (eg: 10.88.0.0/16)

What happened?

I use kubeadm init to setup a first master, but the IP address for the machine is not properly detected for my usage: I want the etcd and api server to advertise an private api address (eg: 10.88.3.60), on a second interface.

I can use kubeadm init --apiserver-advertise-address 10.88.3.60 to bypass ip address detection, but I can't have the same behaviour when using this configuration file and kubeadm init --config=config.yml:

apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: stable
apiServer:
  certSANs:
  - "10.88.3.60"
  extraArgs:
    advertise-address: "10.88.3.60"
    allow-privileged: "true"
    apiserver-count: "3"
controlPlaneEndpoint: "10.88.3.50:7443"

The etcd address is the detected ip address instead of the provided one, and the health check on the api-server is also using the detected ip address.

What you expected to happen?

Setting apiServer.extraArgs.advertise-address in a kubeadm config should have the same effect as --apiserver-advertise-address, or there should be an other way to change the detected ip address globally when using config file.

There is the following configuration change when using a config file with apiServer.extraArgs.advertise-address instead of --apiserver-advertise-address:

diff -ur manifests.with-config/etcd.yaml manifests.with-arg/etcd.yaml
--- manifests.with-config/etcd.yaml	2018-12-23 15:29:23.761519921 +0000
+++ manifests.with-arg/etcd.yaml	2018-12-23 15:31:49.422548222 +0000
@@ -13,15 +13,15 @@
   containers:
   - command:
     - etcd
-    - --advertise-client-urls=https://192.168.122.180:2379
+    - --advertise-client-urls=https://10.88.3.60:2379
     - --cert-file=/etc/kubernetes/pki/etcd/server.crt
     - --client-cert-auth=true
     - --data-dir=/var/lib/etcd
-    - --initial-advertise-peer-urls=https://192.168.122.180:2380
-    - --initial-cluster=master0=https://192.168.122.180:2380
+    - --initial-advertise-peer-urls=https://10.88.3.60:2380
+    - --initial-cluster=master0=https://10.88.3.60:2380
     - --key-file=/etc/kubernetes/pki/etcd/server.key
-    - --listen-client-urls=https://127.0.0.1:2379,https://192.168.122.180:2379
-    - --listen-peer-urls=https://192.168.122.180:2380
+    - --listen-client-urls=https://127.0.0.1:2379,https://10.88.3.60:2379
+    - --listen-peer-urls=https://10.88.3.60:2380
     - --name=master0
     - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
     - --peer-client-cert-auth=true
diff -ur manifests.with-config/kube-apiserver.yaml manifests.with-arg/kube-apiserver.yaml
--- manifests.with-config/kube-apiserver.yaml	2018-12-23 15:29:23.757519892 +0000
+++ manifests.with-arg/kube-apiserver.yaml	2018-12-23 15:31:49.418548194 +0000
@@ -13,8 +13,8 @@
   containers:
   - command:
     - kube-apiserver
-    - --advertise-address=10.88.3.60
     - --authorization-mode=Node,RBAC
+    - --advertise-address=10.88.3.60
     - --allow-privileged=true
     - --client-ca-file=/etc/kubernetes/pki/ca.crt
     - --enable-admission-plugins=NodeRestriction
@@ -44,7 +44,7 @@
     livenessProbe:
       failureThreshold: 8
       httpGet:
-        host: 192.168.122.180
+        host: 10.88.3.60
         path: /healthz
         port: 6443
         scheme: HTTPS

How to reproduce it (as minimally and precisely as possible)?

• Create a VM with 2 NICs, one with a default route and the other with a private ip address and no default route.
• Run kubeadm init --config=<path to config provided in the What happened? part>
• Open the files in /etc/kubernetes/manifests/etcd.yaml and /etc/kubernetes/manifests/ube-apiserver.yaml to observe what was not applied.

[fabriziopandini]

@Congelli501 this is not a defect
apiserver-advertise-address is a node specific parameter (it can change between different control-plane instances) and it has special meaning for kubeadm because it is used in many places, not only for the kube-apiserver. It can be set from the config with a dedicated entry as documented here

extra-args allows to set cluster wide settings (equal among all contr-plane instances) without special meaning to kubeadm (it is a pass through)

I'm closing the issue, but if you need more info let me know
/close

[k8s-ci-robot]

@fabriziopandini: Closing this issue.

In response to this:

@Congelli501 this is not a defect
apiserver-advertise-address is a node specific parameter (it can change between different control-plane instances) and it has special meaning for kubeadm because it is used in many places, not only for the kube-apiserver. It can be set from the config with a dedicated entry as documented here

extra-args allows to set cluster wide settings (equal among all contr-plane instances) without special meaning to kubeadm (it is a pass through)

I'm closing the issue, but if you need more info let me know
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.