GCP Che installation guide does not work #20726

npw3202 · 2021-11-04T01:21:30Z

Describe the bug

I suspect that the GCP installation guide is out of date. I tried following it several times and always got ERR_TOO_MANY_REDIRECTS. I also tried changing from /dashboard/* -> /dashboard/ in the helm patch file and it seems to give another error (404s the manifest.json making me think the nginx is misconfigured)

Che version

7.38@latest

Steps to reproduce

Follow the GCP Che instructions on https://www.eclipse.org/che/docs/che-7/installation-guide/installing-che-on-google-cloud-platform/#creating-a-service-account-secret-on-google-cloud-platform_che

Expected behavior

An eclipse che instancec to be spun up.

Runtime

other (please specify in additional context)

Screenshots

image

Installation method

chectl/latest

Environment

GCE

Eclipse Che Logs

Eclipse Che itself appears to be functioning (installation all went off without a hitch, when I port forward 8080 from the dashboard svc, it appears like the 404'd assets are available).

Additional context

No response

l0rd · 2021-11-04T09:52:16Z

@npw3202 thank you for opening this issue. We are sunsetting installation via the helm chart but have not updated the doc yet. Can you please try installing Che using the operator? You can find some details here.

npw3202 · 2021-11-04T15:59:08Z

@l0rd, Thanks for the follow up, I ran server:delete and then the recommended command using the operator. I got

› Current Kubernetes context: 'gke_***_us-east1-c_prod'
  ✔ Verify Kubernetes API...OK
  ✔ 👀  Looking for an already existing Eclipse Che instance
    ✔ Verify if Eclipse Che is deployed into namespace "eclipse-che"...it is not
  ✔ ✈️  Kubernetes preflight checklist
    ✔ Verify if kubectl is installed
    ✔ Verify remote kubernetes status...done.
    ✔ Check Kubernetes version: Found v1.20.10-gke.1600.
    ✔ Verify domain is set...set to ide.***.com.
    ↓ Check if cluster accessible [skipped]
  ✔ Following Eclipse Che logs
    ✔ Start following Operator logs...done
    ✔ Start following Eclipse Che Server logs...done
    ✔ Start following PostgreSQL logs...done
    ✔ Start following Keycloak logs...done
    ✔ Start following Plug-in Registry logs...done
    ✔ Start following Devfile Registry logs...done
    ✔ Start following Eclipse Che Dashboard logs...done
    ✔ Start following namespace events...done
  ✔ Create Namespace eclipse-che...[Exists]
  ✔ Create Namespace eclipse-che...[Exists]
  ✔ 🏃‍  Running the Eclipse Che operator
    ✔ Create ServiceAccount che-operator in namespace eclipse-che...done.
    ✔ Read Roles and Bindings...done.
    ✔ Creating Roles and Bindings...done.
    ✔ Create CRD checlusters.org.eclipse.che...done.
    ✔ Create backup and restore CRDs...done.
    ✔ Waiting 5 seconds for the new Kubernetes resources to get flushed...done.
    ✔ Create deployment che-operator in namespace eclipse-che...done.
    ✔ Operator pod bootstrap
      ✔ Scheduling...done
      ✔ Downloading images...done
      ✔ Starting...done
    ✔ Prepare Eclipse Che cluster CR...Done.
    ✔ Create the Custom Resource of type checlusters.org.eclipse.che in the namespace eclipse-che...done.
  ✔ ✅  Post installation checklist
    ✔ PostgreSQL pod bootstrap
      ✔ Scheduling...done
      ✔ Downloading images...done
      ✔ Starting...done
    ✔ Keycloak pod bootstrap
      ✔ Scheduling...done
      ✔ Downloading images...done
      ✔ Starting...done
    ✔ Devfile Registry pod bootstrap
      ✔ Scheduling...done
      ✔ Downloading images...done
      ✔ Starting...done
    ✔ Plug-in Registry pod bootstrap
      ✔ Scheduling...done
      ✔ Downloading images...done
      ✔ Starting...done
    ✔ Eclipse Che Dashboard pod bootstrap
      ✔ Scheduling...done
      ✔ Downloading images...done
      ✔ Starting...done
    ✔ Eclipse Che Server pod bootstrap
      ✔ Scheduling...done
      ✔ Downloading images...done
      ✔ Starting...done
    ✔ Eclipse Che status check...done
  ✔ Retrieving Che self-signed CA certificate... commonly trusted certificate is used.
  ✔ Prepare post installation output...done
  ✔ Show important messages
    ✔ Eclipse Che 7.38.1 has been successfully deployed.
    ✔ Documentation             : https://www.eclipse.org/che/docs/
    ✔ -------------------------------------------------------------------------------
    ✔ Users Dashboard           : https://che-eclipse-che.ide.***.com
    ✔ Admin user login          : "admin:admin". NOTE: must change after first login.
    ✔ -------------------------------------------------------------------------------
    ✔ Plug-in Registry          : https://plugin-registry-eclipse-che.ide.***.com/v3/
    ✔ Devfile Registry          : https://devfile-registry-eclipse-che.ide.***.com/
    ✔ -------------------------------------------------------------------------------
    ✔ Identity Provider URL     : https://keycloak-eclipse-che.ide.***.com/auth/
    ✔ Identity Provider login   : "admin:tcuRVWl9upVJ".
    ✔ -------------------------------------------------------------------------------

but when I tried to visit the user's dashboard at https://che-eclipse-che.ide.***.com, I got image suggesting that the nginx proxy might be misconfigured. When I port-forward to the che-dashboard service, the 404'd resources show up on port 8080 (e.g. http://localhost:8080/dashboard/editor.worker.js 200s as expected but https://che-eclipse-che.ide.***.com/dashboard/editor.worker.js 404s). Is the recommended setup steps for the operator's ingress proxy the same as the recommended setup steps in the GCP installation guide? if so, is there any recommended way of debugging this?

tolusha · 2021-11-04T16:54:47Z

pls try: kubectl describe ingress -n eclipse-che
Ingress container logs might shed light on the problem as well.

https://kubernetes.github.io/ingress-nginx/troubleshooting/

npw3202 · 2021-11-04T17:13:06Z

kubectl describe ingress -n eclipse-che yields

Name:             che
Namespace:        eclipse-che
Address:          **.**.***.21
Default backend:  default-http-backend:80 (10.16.10.8:8080)
TLS:
  che-tls terminates che-eclipse-che.ide.***.com
Rules:
  Host                                     Path  Backends
  ----                                     ----  --------
  che-eclipse-che.ide.***.com  
                                           /   che-host:8080 (10.16.7.38:8080)
Annotations:                               che.eclipse.org/managed-annotations-digest: NpIhR7gmyYZJ3KW2WvSw4lBBh-vqSAsF5n83BvCrMeE=
                                           kubernetes.io/ingress.class: nginx
                                           nginx.ingress.kubernetes.io/proxy-connect-timeout: 3600
                                           nginx.ingress.kubernetes.io/proxy-read-timeout: 3600
                                           nginx.ingress.kubernetes.io/ssl-redirect: true
                                           nginx.org/websocket-services: che-host
Events:                                    <none>


Name:             che-dashboard
Namespace:        eclipse-che
Address:          **.**.***.21
Default backend:  default-http-backend:80 (10.16.10.8:8080)
TLS:
  che-tls terminates che-eclipse-che.ide.***.com
Rules:
  Host                                     Path  Backends
  ----                                     ----  --------
  che-eclipse-che.ide.***.com  
                                           /dashboard/   che-dashboard:8080 (10.16.6.157:8080)
Annotations:                               che.eclipse.org/managed-annotations-digest: NpIhR7gmyYZJ3KW2WvSw4lBBh-vqSAsF5n83BvCrMeE=
                                           kubernetes.io/ingress.class: nginx
                                           nginx.ingress.kubernetes.io/proxy-connect-timeout: 3600
                                           nginx.ingress.kubernetes.io/proxy-read-timeout: 3600
                                           nginx.ingress.kubernetes.io/ssl-redirect: true
Events:                                    <none>


Name:             devfile-registry
Namespace:        eclipse-che
Address:          **.**.***.21
Default backend:  default-http-backend:80 (10.16.10.8:8080)
TLS:
  che-tls terminates devfile-registry-eclipse-che.ide.***.com
Rules:
  Host                                                  Path  Backends
  ----                                                  ----  --------
  devfile-registry-eclipse-che.ide.***.com  
                                                        /   devfile-registry:8080 (10.16.6.154:8080)
Annotations:                                            che.eclipse.org/managed-annotations-digest: NpIhR7gmyYZJ3KW2WvSw4lBBh-vqSAsF5n83BvCrMeE=
                                                        kubernetes.io/ingress.class: nginx
                                                        nginx.ingress.kubernetes.io/proxy-connect-timeout: 3600
                                                        nginx.ingress.kubernetes.io/proxy-read-timeout: 3600
                                                        nginx.ingress.kubernetes.io/ssl-redirect: true
Events:                                                 <none>


Name:             keycloak
Namespace:        eclipse-che
Address:          **.**.***.21
Default backend:  default-http-backend:80 (10.16.10.8:8080)
TLS:
  che-tls terminates keycloak-eclipse-che.ide.***.com
Rules:
  Host                                          Path  Backends
  ----                                          ----  --------
  keycloak-eclipse-che.ide.***.com  
                                                /   keycloak:8080 (10.16.6.155:8080)
Annotations:                                    che.eclipse.org/managed-annotations-digest: huexcUXXfXkuznMPAOTnseyJZYt08fRuvuyCj5uAzFE=
                                                kubernetes.io/ingress.class: nginx
                                                nginx.ingress.kubernetes.io/proxy-buffer-size: 16k
                                                nginx.ingress.kubernetes.io/proxy-connect-timeout: 3600
                                                nginx.ingress.kubernetes.io/proxy-read-timeout: 3600
                                                nginx.ingress.kubernetes.io/ssl-redirect: true
Events:                                         <none>


Name:             plugin-registry
Namespace:        eclipse-che
Address:          **.**.***.21
Default backend:  default-http-backend:80 (10.16.10.8:8080)
TLS:
  che-tls terminates plugin-registry-eclipse-che.ide.***.com
Rules:
  Host                                                 Path  Backends
  ----                                                 ----  --------
  plugin-registry-eclipse-che.ide.***.com  
                                                       /   plugin-registry:8080 (10.16.6.156:8080)
Annotations:                                           che.eclipse.org/managed-annotations-digest: NpIhR7gmyYZJ3KW2WvSw4lBBh-vqSAsF5n83BvCrMeE=
                                                       kubernetes.io/ingress.class: nginx
                                                       nginx.ingress.kubernetes.io/proxy-connect-timeout: 3600
                                                       nginx.ingress.kubernetes.io/proxy-read-timeout: 3600
                                                       nginx.ingress.kubernetes.io/ssl-redirect: true
Events:                                                <none>

Dashboard's logs are

Starting Dashboard backend server...
Static server's serving "/public" on 0.0.0.0:8080/dashboard/
Che Dashboard swagger is running on "dashboard/api/swagger".
└── / (GET)
    ├── dashboard (GET)
    │   ├── /
    │   │   ├── * (HEAD)
    │   │   │   * (GET)
    │   │   └── api
    │   │       ├── /swagger/static/* (HEAD)
    │   │       └── /
    │   │           ├── swagger (GET)
    │   │           │   └── / (GET)
    │   │           │       ├── uiConfig (GET)
    │   │           │       ├── initOAuth (GET)
    │   │           │       ├── json (GET)
    │   │           │       ├── yaml (GET)
    │   │           │       ├── static/* (GET)
    │   │           │       └── * (GET)
    │   │           ├── namespace/:/d
    │   │           │   ├── evworkspaces (GET)
    │   │           │   │   └── /:workspaceName (GET)
    │   │           │   └── ockerconfig (GET)
    │   │           └── websocket (GET)
    │   └── /api/namespace/:
    │       ├── /devworkspace
    │       │   ├── s (POST)
    │       │   └── templates (POST)
    │       ├── /devworkspaces/
    │       │   └── :workspaceName (PATCH)
    │       │       :workspaceName (DELETE)
    │       └── /dockerconfig (PUT)
    ├── /
    └── * (OPTIONS)

(node:8) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
Server listening at http://0.0.0.0:8080

When visiting the /dashboard/ path, the nginx's logs give

67.247.235.12 - - [04/Nov/2021:17:04:50 +0000] "GET /dashboard/ HTTP/2.0" 200 842 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36" 346 0.003 [eclipse-che-che-dashboard-8080] [] 10.16.6.157:8080 842 0.003 200 0dcdc4b4a0a9e86a17b56a91bb7df9a2
67.247.235.12 - - [04/Nov/2021:17:04:50 +0000] "GET /dashboard/assets/branding/branding.css HTTP/2.0" 404 548 "https://che-eclipse-che.ide.***.com/dashboard/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36" 121 0.000 [eclipse-che-che-dashboard-8080] [] - - - - 58c324dc21a4989eadca45c27389f36f
67.247.235.12 - - [04/Nov/2021:17:04:50 +0000] "GET /dashboard/assets/branding/loader.svg HTTP/2.0" 404 548 "https://che-eclipse-che.ide.***.com/dashboard/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36" 84 0.000 [eclipse-che-che-dashboard-8080] [] - - - - 846b3d112188e126b8622065338fdf7b
67.247.235.12 - - [04/Nov/2021:17:04:50 +0000] "GET /dashboard/monaco.c5b8fb1575460d099c9a.js HTTP/2.0" 404 548 "https://che-eclipse-che.ide.***.com/dashboard/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36" 52 0.000 [eclipse-che-che-dashboard-8080] [] - - - - c99681b53eb25b6eb62c99bf424a7eac
67.247.235.12 - - [04/Nov/2021:17:04:50 +0000] "GET /dashboard/vendor.140099ebc28dcb96a43e.js HTTP/2.0" 404 548 "https://che-eclipse-che.ide.***.com/dashboard/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36" 42 0.000 [eclipse-che-che-dashboard-8080] [] - - - - dd021b1178917d1a68293243d1db56c9
67.247.235.12 - - [04/Nov/2021:17:04:50 +0000] "GET /dashboard/client.e2b3ef454fcc8ec706b4.js HTTP/2.0" 404 548 "https://che-eclipse-che.ide.***.com/dashboard/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36" 42 0.000 [eclipse-che-che-dashboard-8080] [] - - - - 8a8cb98daf01975650c346c598b3f1c7
67.247.235.12 - - [04/Nov/2021:17:04:50 +0000] "GET /dashboard/service-worker.js HTTP/2.0" 404 548 "https://che-eclipse-che.ide.***.com/dashboard/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36" 33 0.000 [eclipse-che-che-dashboard-8080] [] - - - - 6e725343c4ee268dcb560a483215886d
67.247.235.12 - - [04/Nov/2021:17:04:50 +0000] "GET /dashboard/editor.worker.js HTTP/2.0" 404 548 "https://che-eclipse-che.ide.***.com/dashboard/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36" 33 0.000 [eclipse-che-che-dashboard-8080] [] - - - - 135fe2480c7acc900fdb172e4d87ae36
67.247.235.12 - - [04/Nov/2021:17:04:51 +0000] "GET /dashboard/assets/branding/manifest.json HTTP/2.0" 404 548 "https://che-eclipse-che.ide.***.com/dashboard/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36" 274 0.000 [eclipse-che-che-dashboard-8080] [] - - - - 1fc3c2abbf54dab6ca30d478855a56e1

it appears like the 404'd requests were not routed to 10.16.6.157:8080 because the endpoint described in ingress is /dashboard/ and not /dashboard/. I'll try to change the spec to route to /dashboard/, but I suspect that this will result in a circular redirect (ERR_TOO_MANY_REDIRECTS).

npw3202 · 2021-11-04T17:18:51Z

Ok I think I have a handle of what's going on. For some reason, the default nginx config routes
/dashboard -> the che-dashboard
/dashboard/* -> the che-host.

The che-host redirects
/dashboard/* -> /dashboard/ which redirects to the che-dashboard
Earlier, I believe I accidentally routed (in nginx) /* -> eclipse-host which resulted in the circular redirect.
Let me see if I can manually fix this behavior in the nginx config. I'm curious if this is caused by the artifacts left over from the helm deployment or if the current operator deployment misconfigures nginx...

npw3202 · 2021-11-04T17:33:32Z

I appear to be unable to manually alter the ingress rules (the operator reverts the change before it can be picked up by nginx). I'm unfamiliar with the codebase (I could also be completely wrong about that being the culprit), but at a first glance, could the culprit be this?

tolusha · 2021-11-05T18:04:10Z

I think it is the same issue
#19914

ingress path is /dashboard/ but in case of GCP in must be /dashboard/*

To check this:

scale down operator kubectl scale deployment che-operator --replicas=0 -n eclipse-che
patch ingress kubectl patch ingress/che-dashboard -p '[{"op": "replace", "path": "/spec/rules/0/http/paths/0/path", "value":"/dashboard/*"}]' --type=json -n eclipse-che

The problem is that it is not possible to configure ingress path ^( We need to add such ability

npw3202 · 2021-11-05T19:33:18Z

Thanks for the response, actually it looks like the solution is a little different than suggested. Interestingly enough, that recommended solution didn't work (resulted in an infinite redirect), but changing the ingress path to "/.*" only was also not enough. The annotation

metadata:
    nginx.ingress.kubernetes.io/use-regex: "true"

per https://kubernetes.github.io/ingress-nginx/user-guide/ingress-path-matching/ was also required. Even with GCP, the nginx ingress of "/.*" works (making me think the custom configuration is not needed, but rather, just the addition of the regex annotation is sufficient).

npw3202 · 2021-11-05T19:36:56Z

looks like all is WAI now, lmk if you want me to send a PR to fix this in the operator code / send a PR for documentation fix. Thanks so much for your help.

tolusha · 2021-11-08T07:37:15Z

@npw3202
Yes, pls. That would be nice!

npw3202 · 2021-11-24T20:01:55Z

sorry, somehow this didn't show up on my GitHub issues :-/ I'll have a PR sent by EOD hopefully

tolusha · 2021-11-25T07:23:47Z

@npw3202
Cool! Thank you

tolusha · 2022-01-31T14:37:08Z

@npw3202
Any updates?

themr0c · 2022-06-28T10:06:44Z

The GCP docs have been removed.

npw3202 added the kind/bug Outline of a bug - must adhere to the bug report template. label Nov 4, 2021

che-bot added the status/need-triage An issue that needs to be prioritized by the curator responsible for the triage. See https://github. label Nov 4, 2021

l0rd added severity/P1 Has a major impact to usage or development of the system. area/doc Issues related to documentation and removed status/need-triage An issue that needs to be prioritized by the curator responsible for the triage. See https://github. labels Nov 4, 2021

l0rd added the area/install Issues related to installation, including offline/air gap and initial setup label Nov 4, 2021

tolusha mentioned this issue Nov 11, 2021

Dashboard on AKS not working (too many redirects) #20761

Closed

themr0c closed this as completed Jun 28, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GCP Che installation guide does not work #20726

GCP Che installation guide does not work #20726

npw3202 commented Nov 4, 2021 •

edited

Loading

l0rd commented Nov 4, 2021

npw3202 commented Nov 4, 2021

tolusha commented Nov 4, 2021

npw3202 commented Nov 4, 2021

npw3202 commented Nov 4, 2021 •

edited

Loading

npw3202 commented Nov 4, 2021 •

edited

Loading

tolusha commented Nov 5, 2021

npw3202 commented Nov 5, 2021

npw3202 commented Nov 5, 2021

tolusha commented Nov 8, 2021

npw3202 commented Nov 24, 2021

tolusha commented Nov 25, 2021

tolusha commented Jan 31, 2022

themr0c commented Jun 28, 2022

GCP Che installation guide does not work #20726

GCP Che installation guide does not work #20726

Comments

npw3202 commented Nov 4, 2021 • edited Loading

Describe the bug

Che version

Steps to reproduce

Expected behavior

Runtime

Screenshots

Installation method

Environment

Eclipse Che Logs

Additional context

l0rd commented Nov 4, 2021

npw3202 commented Nov 4, 2021

tolusha commented Nov 4, 2021

npw3202 commented Nov 4, 2021

npw3202 commented Nov 4, 2021 • edited Loading

npw3202 commented Nov 4, 2021 • edited Loading

tolusha commented Nov 5, 2021

npw3202 commented Nov 5, 2021

npw3202 commented Nov 5, 2021

tolusha commented Nov 8, 2021

npw3202 commented Nov 24, 2021

tolusha commented Nov 25, 2021

tolusha commented Jan 31, 2022

themr0c commented Jun 28, 2022

npw3202 commented Nov 4, 2021 •

edited

Loading

npw3202 commented Nov 4, 2021 •

edited

Loading

npw3202 commented Nov 4, 2021 •

edited

Loading