'Workspace is not running' page appeared even when the workspace should be running

[dkwon17]

Describe the bug

I started a workspace, but instead of opening the editor, the Your workspace is not running page opened instead:

Screen.Recording.2022-10-31.at.4.50.23.PM.mov

This issue does not happen does not happen often (from my experience, maybe once in 50 times), but it is worth investigating.

IIRC if the workspace service cannot be reached and the workspace url is accessed, we will see this Your workspace is not running page. It could be worth investigating if the traefik retry middleware can help in this case, by retrying when the workspace service cannot be accessed multiple times before giving up and redirecting the an error/info page.

Che version

next (development version)

Steps to reproduce

Not able to reproduce consistently, but the issue happened when I was starting an existing workspace.

Expected behavior

The Your workspace is not running page should not be displayed. Instead, we should be seeing the editor since the workspace has started successfully.

Runtime

OpenShift

Screenshots

No response

Installation method

chectl/next

Environment

macOS

Eclipse Che Logs

No response

Additional context

I experienced this issue on the dogfooding cluster

[dkwon17]

After some more investigation the problem seems to be happening because the devworkspace's status becomes:

Starting -> Running -> Starting* -> Running

where the Starting* phase typically shouldn't happen.

Watching the issue video, when the workspace is starting, the devworkspace's status.phase is Starting. At the 0:21 mark, we can see that the status.phase becomes Running for less than 1 second, and then goes back to Starting again.

Then, going back to the dashboard at the 0:28 mark, the workspace's status is back to Running , as indicated by the green symbol beside the workspace name.

The DWO sets the devworkspace's phase to Running only if the workspace service health check returns a 2xx or 4xx error code as seen here (serverRunning boolean is only true if the workspace service's health check endpoint returns a 2xx or 4xx error code here).

Since the devworkspace's phase goes temporarily from Running to Starting* , it seems like a 5xx error code is being returned right when the dashboard tries to access the workspace url, causing the 'Workspace is not running' page to appear.

Why the workspace goes from Running to Starting* in the first place, is still a mystery however.

There is already a dashboard-side check to mitigate this issue, however, it did not prevent it from happening in the video.
On the dashboard side, before creating/displaying the 'Workspace is not running' page, the dashboard checks if the workspace's status returns to Running here . If yes, the dashboard directs the browser to the workspace URL.

[ibuziuk]

@dkwon17 wondering if we can close this issue in favor of #22008 ?

[dkwon17]

@ibuziuk yes, sounds good