jshell Eval Exception when accessing Kernel Id

[prisaparoy]

Hi,
It seems like connection between Notebook and Kernel is not getting established

[I 2023-04-20 23:52:56.487 ServerApp] Restoring connection for a1009026-6a24-4a3c-a81d-43f6627f185f:f76ffc26-c2fb-4efa-b808-a078d5efb5ce
INFO [ main] JettyServletWebServerFactory : Server initialized with port: 0
INFO [ main] Server : jetty-10.0.13; built: 2022-12-07T20:13:20.134Z; git: 1c2636ea05c0ca8de1ffd6ca7f3a98ac084c766d; jvm 20+36-2344
INFO [ main] application : Initializing Spring embedded WebApplicationContext
INFO [ main] ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 443 ms
INFO [ main] DefaultSessionIdManager : Session workerName=node0
INFO [ main] ContextHandler : Started o.s.b.w.e.j.JettyEmbeddedWebAppContext@51b77cdf{application,/,[file:///tmp/jetty-docbase.0.6184187775426036712/],AVAILABLE}
INFO [ main] Server : Started Server@40d60f2{STARTING}[10.0.13,sto=0] @2447ms
INFO [ main] application : Initializing Spring DispatcherServlet 'dispatcherServlet'
INFO [ main] DispatcherServlet : Initializing Servlet 'dispatcherServlet'
INFO [ main] DispatcherServlet : Completed initialization in 0 ms
INFO [ main] AbstractConnector : Started ServerConnector@ea52184{HTTP/1.1, (http/1.1)}{0.0.0.0:39131}
INFO [ main] JettyWebServer : Jetty started on port(s) 39131 (http/1.1) with context path '/'
INFO [ main] Server : Kernel a1009026-6a24-4a3c-a81d-43f6627f185f
INFO [ main] Server : Connected to {
"shell_port" : 54297,
"iopub_port" : 54557,
"stdin_port" : 38129,
"control_port" : 40219,
"hb_port" : 37385,
"ip" : "127.0.0.1",
"key" : "07ebde7c-65b0f1581c5df5905fae2a90",
"transport" : "tcp",
"signature_scheme" : "hmac-sha256",
"kernel_name" : "ganymede-2.1.1-java-20-spark",
"pid" : 2430046,
"port" : 34863
}
INFO [pool-1-thread-5] Dispatcher : Bound REP tcp://127.0.0.1:37385
INFO [pool-1-thread-2] Dispatcher : Bound ROUTER tcp://127.0.0.1:40219
INFO [pool-1-thread-3] Dispatcher : Bound PUB tcp://127.0.0.1:54557
INFO [pool-1-thread-4] Dispatcher : Bound ROUTER tcp://127.0.0.1:38129
INFO [pool-1-thread-1] Dispatcher : Bound ROUTER tcp://127.0.0.1:54297
WARN [pool-1-thread-1] Server : Ignoring non-request comm_open
WARN [pool-1-thread-1] Server : Ignoring non-request comm_msg
[W 2023-04-20 23:53:22.483 ServerApp] wrote error: 'Forbidden'
Traceback (most recent call last):
File "/home/prisaparoy/mambaforge/envs/jupyter3.6/lib/python3.10/site-packages/tornado/web.py", line 1711, in _execute
result = method(*self.path_args, **self.path_kwargs)
File "/home/prisaparoy/mambaforge/envs/jupyter3.6/lib/python3.10/site-packages/tornado/web.py", line 3196, in wrapper
url = self.get_login_url()
File "/home/prisaparoy/mambaforge/envs/jupyter3.6/lib/python3.10/site-packages/jupyter_server/base/handlers.py", line 747, in get_login_url
raise web.HTTPError(403)
tornado.web.HTTPError: HTTP 403: Forbidden
[W 2023-04-20 23:53:22.484 ServerApp] 403 GET /api/kernels/a1009026-6a24-4a3c-a81d-43f6627f185f (@127.0.0.1) 2.66ms referer=None
WARN [pool-1-thread-1] Shell : jdk.jshell.EvalException
WARN [pool-1-thread-1] Shell : at ganymede.notebook.NotebookContext.(NotebookContext.java:93)
WARN [pool-1-thread-1] Shell : at ganymede.notebook.Notebook.newNotebookContext(Notebook.java:45)
WARN [pool-1-thread-1] Shell : at .do_it$Aux(#1:1)
WARN [pool-1-thread-1] Shell : at .(#1:1)
WARN [pool-1-thread-1] Shell : Caused by: jdk.jshell.EvalException: apiKernelsKernelIdGet call failed with: 403 - {"message": "Forbidden", "reason": null}
WARN [pool-1-thread-1] Shell : at ganymede.jupyter.notebook.api.KernelsApi.getApiException(KernelsApi.java:71)
WARN [pool-1-thread-1] Shell : at ganymede.jupyter.notebook.api.KernelsApi.apiKernelsKernelIdGetWithHttpInfo(KernelsApi.java:263)
WARN [pool-1-thread-1] Shell : at ganymede.jupyter.notebook.api.KernelsApi.apiKernelsKernelIdGet(KernelsApi.java:239)
WARN [pool-1-thread-1] Shell : at ganymede.jupyter.NotebookServicesClient.getKernel(NotebookServicesClient.java:140)
WARN [pool-1-thread-1] Shell : at ganymede.notebook.NotebookContext.(NotebookContext.java:91)
WARN [pool-1-thread-1] Shell : ... 3 more

Environment:
Jupyterlab 3.6 Inside Conda environment.
Python: 3.10.8
Java: 11 or 20
OS: Ubuntu 22.04

Do you see anything about this setup that would lead to this error?

[Thejayman77]

I was getting pretty much the same thing and after messing around with everything from python to java to different server OSes, I finally found that a hashed password is what causes the connection to break. Once I removed my hashed password and relied on the token instead, the connection worked fine. After adding a password in again, it did the same as above.

Anyhow, perhaps this might help find out what the cause is. Let me know if it helps you out as well or not so perhaps I can dig in a little bit to see what might be going on. At least it's a workaround for now if you can use a tokenized login.

[prisaparoy]

Yes, removing hashed password fixed connection issue for now. Good find and thank you!.

[Thejayman77]

No problem, glad that helped out!