creating voyager ingress on k8s version 1.20 is failing

[sankarpn]

Trying to create voyager ingress on k8s version 1.20 in Oracle Linux Cloud Native Environment 1.3
https://docs.oracle.com/en/operating-systems/olcne/index.html

The voyager service is not getting created.
Here is the complete reproduction steps.

This k8s cluster is running CRIO runtime

bash $ kubectl get nodes -o wide
NAME      STATUS   ROLES                  AGE   VERSION         INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                  KERNEL-VERSION                     CONTAINER-RUNTIME
olcne-2   Ready    control-plane,master   9d    v1.20.6+1.el7   ***********   <none>        Oracle Linux Server 7.9   5.4.17-2102.200.13.el7uek.x86_64   cri-o://1.20.2
olcne-3   Ready    <none>                 9d    v1.20.6+1.el7   ***********   <none>        Oracle Linux Server 7.9   5.4.17-2102.200.13.el7uek.x86_64   cri-o://1.20.2

base $ helm repo add appscode https://charts.appscode.com/stable/
"appscode" has been added to your repositories

bash $ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "ingress-nginx" chart repository
...Successfully got an update from the "weblogic-operator" chart repository
...Successfully got an update from the "traefik" chart repository
...Successfully got an update from the "appscode" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈ Happy Helming!⎈


bash $ helm install voyager-release-dpam appscode/voyager --namespace ns-dpam  --version 12.0.0 --set "apiserver.enableValidatingWebhook=false" --set "cloudProvider=baremetal"
NAME: voyager-release-dpam
LAST DEPLOYED: Fri Jul  9 17:36:09 2021
NAMESPACE: ns-dpam
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Set cloudProvider for installing Voyager

To verify that Voyager has started, run:

  kubectl get deployment --namespace ns-dpam -l "app.kubernetes.io/name=voyager,app.kubernetes.io/instance=voyager-release-dpam"

bash $ cat voyager.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.appscode.com/affinity: cookie
    ingress.appscode.com/type: NodePort
    kubernetes.io/ingress.class: voyager
  name: coherence-managed-domain1-ingress-host-routing
  namespace: ns-ejvn
spec:
  rules:
  - host: coherence-managed-domain1.ns-ejvn.cluster-1.test
    http:
      paths:
      - backend:
          serviceName: coherence-managed-domain1-cluster-cluster-1
          servicePort: 8001
  - host: coherence-managed-domain1.ns-ejvn.cluster-2.test
    http:
      paths:
      - backend:
          serviceName: coherence-managed-domain1-cluster-cluster-2
          servicePort: 8001


bash $ kubectl create -f voyager.yaml
Warning: networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
ingress.networking.k8s.io/coherence-managed-domain1-ingress-host-routing created


bash $ kubectl get services -n ns-ejvn -w
NAME                                                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)              AGE
coherence-managed-domain1-admin-server                ClusterIP   None            <none>        30012/TCP,7001/TCP   108m
coherence-managed-domain1-admin-server-ext            NodePort    10.109.3.30     <none>        7001:30229/TCP       108m
coherence-managed-domain1-cluster-1-managed-server1   ClusterIP   None            <none>        8001/TCP             107m
coherence-managed-domain1-cluster-1-managed-server2   ClusterIP   None            <none>        8001/TCP             107m
coherence-managed-domain1-cluster-2-managed-server1   ClusterIP   None            <none>        8001/TCP             107m
coherence-managed-domain1-cluster-2-managed-server2   ClusterIP   None            <none>        8001/TCP             107m
coherence-managed-domain1-cluster-cluster-1           ClusterIP   10.103.130.66   <none>        8001/TCP             107m
coherence-managed-domain1-cluster-cluster-2           ClusterIP   10.102.211.96   <none>        8001/TCP             107m
voyager-coherence-managed-domain1-ingress-host-routing   NodePort    10.98.65.190    <none>        80:32420/TCP         0s
voyager-coherence-managed-domain1-ingress-host-routing   NodePort    10.98.65.190    <none>        80:32420/TCP         0s
voyager-coherence-managed-domain1-ingress-host-routing   NodePort    10.96.118.25    <none>        80:30313/TCP         0s
voyager-coherence-managed-domain1-ingress-host-routing   NodePort    10.96.118.25    <none>        80:30313/TCP         0s
voyager-coherence-managed-domain1-ingress-host-routing   NodePort    10.101.125.31   <none>        80:30898/TCP         0s
voyager-coherence-managed-domain1-ingress-host-routing   NodePort    10.101.125.31   <none>        80:30898/TCP         0s

voyager operator logs shows this messages

bash $ kubectl logs -n ns-dpam pod/voyager-release-dpam-6f5c48dc55-kkrmg

I0709 17:38:06.065975 1 services.go:74] Add/Delete/Update of offshoot service ns-ejvn/voyager-coherence-managed-domain1-ingress-host-routing, Ingress ns-ejvn/coherence-managed-domain1-ingress-host-routing re-queued for update
E0709 17:38:11.057137 1 worker.go:92] Failed to process key ns-ejvn/coherence-managed-domain1-ingress-host-routing. Reason: services "voyager-coherence-managed-domain1-ingress-host-routing" not found
I0709 17:38:11.057487 1 worker.go:96] Error syncing key ns-ejvn/coherence-managed-domain1-ingress-host-routing: services "voyager-coherence-managed-domain1-ingress-host-routing" not found
I0709 17:38:11.057735 1 nodeport.go:91] [2bba0482-3244-4e54-a4b8-d0857fd48f73] Initializing cloud manager for provider baremetal
I0709 17:38:11.057844 1 nodeport.go:119] [2bba0482-3244-4e54-a4b8-d0857fd48f73] No cloud manager found for provider baremetal
I0709 17:38:11.057868 1 ingresses.go:128] Sync/Add/Update for ingress ns-ejvn/coherence-managed-domain1-ingress-host-routing
I0709 17:38:11.061296 1 service.go:35] Creating Service ns-ejvn/voyager-coherence-managed-domain1-ingress-host-routing.
I0709 17:38:11.078513 1 services.go:55] Sync/Add/Update for Service voyager-coherence-managed-domain1-ingress-host-routing
I0709 17:38:11.091023 1 endpoints.go:52] Sync/Add/Update for Endpoints ns-ejvn/voyager-coherence-managed-domain1-ingress-host-routing
W0709 17:38:11.095246 1 services.go:43] Service ns-ejvn/voyager-coherence-managed-domain1-ingress-host-routing does not exist anymore

kube-apiserver shows these messages

bash $ kubectl logs -n kube-system pod/kube-apiserver-olcne-2

W0709 17:14:25.196826 1 controller.go:142] slow openapi aggregation of "certificates.voyager.appscode.com": 1.017908073s
E0709 17:14:26.070779 1 customresource_handler.go:669] error building openapi models for ingresses.voyager.appscode.com: ERROR $root.definitions.com.appscode.voyager.v1beta1.Ingress.properties.spec.properties.backend.properties.servicePort has invalid property: anyOf
ERROR $root.definitions.com.appscode.voyager.v1beta1.Ingress.properties.spec.properties.frontendRules.items..properties.port has invalid property: anyOf
ERROR $root.definitions.com.appscode.voyager.v1beta1.Ingress.properties.spec.properties.livenessProbe.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.appscode.voyager.v1beta1.Ingress.properties.spec.properties.livenessProbe.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.appscode.voyager.v1beta1.Ingress.properties.spec.properties.readinessProbe.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.appscode.voyager.v1beta1.Ingress.properties.spec.properties.readinessProbe.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.appscode.voyager.v1beta1.Ingress.properties.spec.properties.rules.items..properties.http.properties.nodePort has invalid property: anyOf
ERROR $root.definitions.com.appscode.voyager.v1beta1.Ingress.properties.spec.properties.rules.items..properties.http.properties.paths.items..properties.backend.properties.servicePort has invalid property: anyOf
ERROR $root.definitions.com.appscode.voyager.v1beta1.Ingress.properties.spec.properties.rules.items..properties.http.properties.port has invalid property: anyOf
ERROR $root.definitions.com.appscode.voyager.v1beta1.Ingress.properties.spec.properties.rules.items..properties.tcp.properties.backend.properties.servicePort has invalid property: anyOf
ERROR $root.definitions.com.appscode.voyager.v1beta1.Ingress.properties.spec.properties.rules.items..properties.tcp.properties.nodePort has invalid property: anyOf
ERROR $root.definitions.com.appscode.voyager.v1beta1.Ingress.properties.spec.properties.rules.items..properties.tcp.properties.port has invalid property: anyOf

[tamalsaha]

@sankarpn , please try the https://github.com/voyagermesh/voyager/releases/tag/v13.0.0-rc.0 release. That should work.

[sankarpn]

Its still not working, I tried both https://artifacthub.io/packages/helm/appscode/voyager/13.0.0-beta.1 and https://artifacthub.io/packages/helm/appscode/voyager/2021.4.24-rc.0

[tamalsaha]

@sankarpn, if you are interested in acquiring license for the Voyager Enterprise, please fill up the form here: https://appscode.com/contact/ . We have stopped doing free work for other businesses.

[sankarpn]

@tamalsaha We don't use voyager in production. We test and certify that voyager works with our Oracle WebLogic k8s operator in different environments. There are a lot of customers will be using voyager when we certify.

[tamalsaha]

@sankarpn , that's exciting. Could we set up a call to get a better understanding?

[tamalsaha]

My can reach me by email tamal AT appscode.com .

[sankarpn]

I tried to install version 2021.4.24-rc.0 using the configuration values from https://artifacthub.io/packages/helm/appscode/voyager/2021.4.24-rc.0 , looks like the "operator.tag" doesn't set the release version.

helm install voyager-release50-huha appscode/voyager --namespace ns-huha --set "operator.tag=v13.0.0-rc.0" --set "apiserver.enableValidatingWebhook=true" --set "cloudProvider=baremetal"

As you can see from the following describe it still uses the v12.0.0 version

[opc@olcne-1 ~]$ kubectl describe -n ns-huha pod/voyager-release50-huha-68c5c66d-l5w8m
Name: voyager-release50-huha-68c5c66d-l5w8m
Namespace: ns-huha
Priority: 0
Node: olcne-3/10.0.3.76
Start Time: Fri, 03 Sep 2021 18:07:48 +0000
Labels: app.kubernetes.io/instance=voyager-release50-huha
app.kubernetes.io/name=voyager
pod-template-hash=68c5c66d
Annotations: checksum/apiregistration.yaml: 351fa9985c13336fe4fd47127ed5f6f165a1c79e1c60d6a1296709b9331a4234
Status: Running
IP: 10.244.1.99
IPs:
IP: 10.244.1.99
Controlled By: ReplicaSet/voyager-release50-huha-68c5c66d
Containers:
voyager:
Container ID: cri-o://59d8bc2d720fc76c096536acc022d52c2669c5e1d2f7889e5fe6e7f2f99627d1
Image: appscode/voyager:v12.0.0
Image ID: docker.io/appscode/voyager@sha256:1569c90b098c301fde05efeab508c5adc44c63690e091e9db55e6d362de04adb
Ports: 8443/TCP, 56791/TCP
Host Ports: 0/TCP, 0/TCP
Args:
run
--cloud-provider=baremetal
--cloud-config=
--v=3
--ingress-class=
--operator-service=voyager-release50-huha
--docker-registry=appscode
--haproxy-image-tag=1.9.15-v12.0.0-alpine
--haproxy-image-repository=haproxy
--exporter-image-tag=v12.0.0
--exporter-image-repository=voyager
--secure-port=8443
--audit-log-path=-
--tls-cert-file=/var/serving-cert/tls.crt
--tls-private-key-file=/var/serving-cert/tls.key
--enable-validating-webhook=true
--bypass-validating-webhook-xray=false
--use-kubeapiserver-fqdn-for-aks=true
--enable-analytics=true
State: Running
Started: Fri, 03 Sep 2021 18:07:49 +0000
Ready: True
Restart Count: 0
Environment:
MY_POD_NAME: voyager-release50-huha-68c5c66d-l5w8m (v1:metadata.name)
MY_POD_NAMESPACE: ns-huha (v1:metadata.namespace)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from voyager-release50-huha-token-47xvl (ro)
/var/serving-cert from serving-cert (rw)

I had to use the voyager.tag to correctly install the version desired.

helm install voyager-release50-huha appscode/voyager --namespace ns-huha --set "voyager.tag=v13.0.0-rc.0" --set "apiserver.enableValidatingWebhook=true" --set "cloudProvider=baremetal"

[opc@olcne-1 ~]$ kubectl describe -n ns-huha pod/voyager-release50-huha-7f46b9fb6f-9p6lx
Name: voyager-release50-huha-7f46b9fb6f-9p6lx
Namespace: ns-huha
Priority: 0
Node: olcne-3/10.0.3.76
Start Time: Fri, 03 Sep 2021 18:10:39 +0000
Labels: app.kubernetes.io/instance=voyager-release50-huha
app.kubernetes.io/name=voyager
pod-template-hash=7f46b9fb6f
Annotations: checksum/apiregistration.yaml: de5e98866b6881061ef385371c49e7e22da221ac39955760e8da95c6182f9394
Status: Running
IP: 10.244.1.101
IPs:
IP: 10.244.1.101
Controlled By: ReplicaSet/voyager-release50-huha-7f46b9fb6f
Containers:
voyager:
Container ID: cri-o://71c6fcb89bbfbfef3bb735e198a6029eceb4ec36c3f8dfcd9a36c66e9707f7f4
Image: appscode/voyager:v13.0.0-rc.0
Image ID: docker.io/appscode/voyager@sha256:6d35a828ef929c04c5cda70c8013389a2de78035d04bc10c339b585588b5abb0
Ports: 8443/TCP, 56791/TCP
Host Ports: 0/TCP, 0/TCP
Args:
run
--cloud-provider=baremetal
--cloud-config=
--v=3
--ingress-class=
--operator-service=voyager-release50-huha
--docker-registry=appscode
--haproxy-image-tag=1.9.15-v12.0.0-alpine
--haproxy-image-repository=haproxy
--exporter-image-tag=v13.0.0-rc.0
--exporter-image-repository=voyager
--secure-port=8443
--audit-log-path=-
--tls-cert-file=/var/serving-cert/tls.crt
--tls-private-key-file=/var/serving-cert/tls.key
--enable-validating-webhook=true
--bypass-validating-webhook-xray=false
--use-kubeapiserver-fqdn-for-aks=true
--enable-analytics=true

But again look at the --haproxy-image-tag=1.9.15-v12.0.0-alpine which is not matching with the version 1.9.15-v13.0.0-rc.0-alpine.

Also the operator pod kept crashing.

2m11s Normal Created pod/voyager-release50-huha-7f46b9fb6f-9p6lx Created container voyager
2m11s Normal Pulled pod/voyager-release50-huha-7f46b9fb6f-9p6lx Container image "appscode/voyager:v13.0.0-rc.0" already present on machine
2m7s Warning AdmissionWebhookNotActivated deployment/voyager-release50-huha timed out waiting for the condition
99s Warning BackOff pod/voyager-release50-huha-7f46b9fb6f-9p6lx Back-off restarting failed container

[tamalsaha]

Hello, We have released https://blog.byte.builders/post/voyager-v2021.09.15/ . Can you please check if this works with cri-o?