minikube doesn't start in VM with no virtualisation support

[jetojedno]

I'm not sure if this is a bug or a feature request.

We have available corporate VMs which have virtualisation support in the (virtual) CPU disabled. I have tried for a week to get minikube with docker running, and each time I fix one blocker, there's another waiting.

This should be easy. :-) But it's not.

Please include this scenario in the test sets for minikube.

I'm working on Ubuntu 16.04, with the latest (as of yesterday) minikube.

[tstromberg]

We do actually test this regularly in our integration tests, using --vm-driver=none. Could you be a bit more specific in what you are experiencing?

[afbjorklund]

This scenario seems to be more and more common (using a VM instead of a laptop), and probably needs some targeted setup documentation...

[jetojedno]

If it makes things any clearer ... I've got this working yesterday for me, using info from a number of sources.

This is the script I use:

## start minikube on bare VM
# from: https://github.com/kubernetes/minikube/blob/master/docs/vmdriver-none.md
# from: https://github.com/Kingonion/minikube-on-linux
# from: https://medium.com/@nieldw/running-minikube-with-vm-driver-none-47de91eab84c
# *** Hmm. a lot of mixed instructions

sudo echo 'Just checking sudo works' > /dev/null || exit $?
sudo systemctl is-active docker >/dev/null || sudo systemctl status docker || exit $?

which minikube || (
  curl -Lo minikube \
    https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 \
    && chmod +x minikube \
    && sudo install minikube /usr/local/bin/ \
    && rm minikube || exit $?
)

which kubectl || (
  KUBE_VERSION=$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt) \
  && [ -n "$KUBE_VERSION" ] || exit $?
  curl -Lo kubectl \
    https://storage.googleapis.com/kubernetes-release/release/$KUBE_VERSION/bin/linux/amd64/kubectl \
    && chmod +x kubectl \
    && sudo install kubectl /usr/local/bin/ \
    && rm kubectl || exit $?
)

echo "minikube: $(minikube version)" || exit $?
echo "kubectl: $(kubectl version --client=true --short=true)" || exit $?
echo "docker: $(docker info | grep 'Server Version')" || exit $?
MINIKUBE_VERSION=$( minikube version | awk -F ':' '{ print $2 }' | tr -d '[:blank:]' ) \
  && [ -n "$MINIKUBE_VERSION" ] || exit $?
KUBE_VERSION=$( kubectl version --client=true --short=true | awk -F ':' '{ print $2 }' | tr -d '[:blank:]' ) \
  && [ -n "$KUBE_VERSION" ] || exit $?

export MINIKUBE_WANTUPDATENOTIFICATION=false
export MINIKUBE_WANTREPORTERRORPROMPT=false
export MINIKUBE_HOME="$HOME"
export CHANGE_MINIKUBE_NONE_USER=true
export KUBECONFIG="$HOME/.kube/config"
DOCKER_CONFIG="$HOME/.docker/config.json"
API_SERVER_IP='127.0.0.1'
SERVICE_CLUSTER_IP_RANGE='172.19.0.0/16'

mkdir -p "$HOME/.kube" "$HOME/.minikube" "$HOME/.docker"
[ -f "$KUBECONFIG" ] || touch "$KUBECONFIG" || exit $?
[ -f "$DOCKER_CONFIG" ] || echo '{}' > "$DOCKER_CONFIG" || exit $?

minikube config set vm-driver none
minikube config set kubernetes-version "${KUBE_VERSION}"

cgroup_driver=$(docker info 2>/dev/null | grep 'Cgroup Driver' | awk -F ':' '{ print $2 }' | tr -d '[:blank:]') \
  && [ -n "$cgroup_driver" ] || exit $?

sudo -E minikube start \
  --vm-driver=none \
  --kubernetes-version "${KUBE_VERSION}" \
  --extra-config kubelet.cgroup-driver="${cgroup_driver}" \
  --apiserver-ips "$API_SERVER_IP" --apiserver-name localhost \
  --service-cluster-ip-range "$SERVICE_CLUSTER_IP_RANGE"
sudo -E minikube stop
sudo chown --reference="$HOME" -R "$HOME/.kube" "$HOME/.minikube" "$HOME/.docker"

MINIKUBE_IP=$( minikube ip | tr -d '[:blank:]' ) \
  && [ -n "$MINIKUBE_IP" ] && [ "$MINIKUBE_IP" != "$API_SERVER_IP" ] \
  && sed "s|/$( echo -n $MINIKUBE_IP | sed 's/\./\\./g' ):|/$API_SERVER_IP:|" "$KUBECONFIG" > /tmp/kube_config \
  && mv /tmp/kube_config "$KUBECONFIG"
sudo -E minikube start \
  --vm-driver=none \
  --kubernetes-version "${KUBE_VERSION}" \
  --extra-config kubelet.cgroup-driver="${cgroup_driver}" \
  --apiserver-ips "$API_SERVER_IP" --apiserver-name localhost \
  --service-cluster-ip-range "$SERVICE_CLUSTER_IP_RANGE"

kubectl cluster-info
echo
kubectl get namespaces
echo
kubectl get services --all-namespaces
echo
kubectl get pods -n kube-system
echo

[afbjorklund]

Why are you starting it twice ? And I wonder why the cgroup driver needs tweaking like that, I was hoping for it to do the right thing out-of-the-box. But maybe it is needed, don't really this setup myself.

[tstromberg]

So that we can give this bug a better title - what error messages were you seeing?

[jetojedno]

Why are you starting it twice ? And I wonder why the cgroup driver needs tweaking like that, I was hoping for it to do the right thing out-of-the-box. But maybe it is needed, don't really this setup myself.

I agree about the cgroup driver ... I would have expected minikube to detect docker's cgroup driver ... but it doesn't seem to.

I start it twice as, between the two, I adjust the IP address for the APIserver in the config file to localhost (127.0.0.1) - as was specified on the command line options. The second start is after the config adjustment and, in part, checks that the adjustment was done correctly.

[jetojedno]

The error message firstly the cgroup driver (kubelet failed with kubelet cgroup driver: “cgroupfs” is different from docker cgroup driver: “systemd”), then a timeout connecting to the auto-created nodes: the API server (I couldn't get it to work using the minikube default of the VM's external IP address) and then the proxy.

The timeout error message:

Waiting for pods: apiserver proxy
!   Error restarting cluster: wait: waiting for k8s-app=kube-proxy: timed out waiting for the condition

I hacked together this script at that point, so can't say if there would be further errors in the sequence ...

[jetojedno]

vm-driver=none: kubelet fails to come up when docker_daemon.json has native.cgroupdriver=systemd

[jetojedno]

I should add that I'm ssh'ing into the VM to do all this (and to control the VM) so access over the network is neither needed nor desirable. Obviously, others may have different requirements, once minikube is deployed and running.

[jetojedno]

... thinking about this, and looking at what I've written, an ideal solution would be to create, within docker, both a subnet range which is exclusive to minikube, and within that, a bridge network for connecting the created minikube docker nodes ... with a gateway IP address which can be used by kubectl to talk to all the pods. This would be a direct equivalent of the VM networking and access could easily be constrained by iptables as it is would be with a created minikube VM.

My knowledge of minikube & docker isn't adequate to do this (I'm using minikube to follow a course about kubernetes) but I can see the objective. I'm learning more than I expected, although this is slow going. :-)

If you can advise about suitable options or commands or changes, I can hack the script to make it do this ...

The first thing I will do is add a couple of options to the script (subnet & bridge CIDR & gateway IP address) and code to read these from /etc/hosts and /etc/networks (or extant docker network stuff) if not specified. The latter allows for restart without respecifying all options.

[tstromberg]

This issue appears to be a duplicate of #4172, so I will close this one so that we may centralize the content relating to the issue. If you feel that this issue is not in fact a duplicate, please feel free to re-open it.

Thank you for reporting this!