api/shutdown does not shutdown running kernels

[martinRenou]

Description

The api/shutdown end point is not shutting down the kernel sessions. This leaves the kernel processes hanging if they are not properly shut down before trying to shutdown the server.

This forces us to do the following in JupyterLab prior to sending POST api/shutdown: https://github.com/jupyterlab/jupyterlab/pull/10843/files

Expected behavior

It might be nicer for the api/shutdown handler to first shut down the kernels before shutting down the server.

Context

This would be useful in contexts where e.g. JupyterLab is embedded in an electron app, so that we can overwrite the "close app" button behavior so that it sends the api/shutdown request only, not having to shut down all kernels manually.

[kevin-bates]

Hi @martinRenou - the intention of the shutdown request is that kernels (and terminals) be shutdown. This is the case in Notebook and in Server. However, I am seeing some intermittency and interference when running against Server.

Here's the console output of each:
Notebook:

I 07:36:49.221 LabApp] Shutting down on /api/shutdown request.
[I 07:36:49.229 LabApp] Shutting down 1 kernel
[I 07:36:49.230 LabApp] Kernel shutdown: 09772cab-e7a0-4b40-b5a8-1dbc60ca8c7c
[I 07:36:49.433 LabApp] Shutting down 0 terminals

Server:

-[I 2021-08-31 07:52:26.955 ServerApp] Kernel shutdown: 27cec4d3-4bf3-4cc3-99d7-67b837cfe50f
|[I 2021-08-31 07:52:27.192 ServerApp] Shutting down on /api/shutdown request.
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/opt/anaconda3/envs/elyra-dev/lib/python3.8/site-packages/jupyterlab_server/process.py", line 186, in _cleanup
    proc.terminate()
  File "/opt/anaconda3/envs/elyra-dev/lib/python3.8/site-packages/jupyterlab_server/process.py", line 133, in terminate
    Process._procs.remove(self)
  File "/opt/anaconda3/envs/elyra-dev/lib/python3.8/_weakrefset.py", line 109, in remove
    self.data.remove(ref(item))
KeyError: <weakref at 0x7fcf665867c0; to 'ProgressProcess' at 0x7fcf6662c580>

One of the recent changes was to allow server extensions to hook the shutdown, and I suspect that path is encountering an issue. However, if I disable all server extensions except lab itself, I still get the KeyError regarding the weak reference.

If I revert the server to its 1.9 release so that it does not include the referenced PR and I revert Lab to 3.1.1 so that it doesn't include the change you referenced above, I get the following output:

/[I 2021-08-31 08:20:21.806 ServerApp] Shutting down on /api/shutdown request.
[I 2021-08-31 08:20:21.810 ServerApp] Shutting down 1 kernel
[I 2021-08-31 08:20:21.811 ServerApp] Kernel shutdown: 8d665977-cd3c-4f65-aaed-dd22bc5d328b
\[I 2021-08-31 08:20:22.024 ServerApp] Shutting down 0 terminals
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/opt/anaconda3/envs/elyra-dev/lib/python3.8/site-packages/jupyterlab_server/process.py", line 186, in _cleanup
    proc.terminate()
  File "/opt/anaconda3/envs/elyra-dev/lib/python3.8/site-packages/jupyterlab_server/process.py", line 133, in terminate
    Process._procs.remove(self)
  File "/opt/anaconda3/envs/elyra-dev/lib/python3.8/_weakrefset.py", line 109, in remove
    self.data.remove(ref(item))
KeyError: <weakref at 0x7f85f0799a90; to 'ProgressProcess' at 0x7f85f06938e0>

If I keep the server down-level (1.9) but retain the changes made two weeks ago, I see this:

-[I 2021-08-31 08:13:54.977 ServerApp] Kernel shutdown: c29e5328-6501-4a5c-9805-1686ad244616
-[I 2021-08-31 08:13:55.303 ServerApp] Shutting down on /api/shutdown request.
[I 2021-08-31 08:13:55.310 ServerApp] Shutting down 0 kernels
[I 2021-08-31 08:13:55.311 ServerApp] Shutting down 0 terminals

which is consistent - just that the lab change is interfering with the server's more "graceful" attempt and, thus, ,0 kernels or terminals will be shut down.

This is a regression issue.

What output are you seeing when calling /api/shutdown? I'm curious if you also see the weakref issue in jupyterlab_server.

[martinRenou]

the intention of the shutdown request is that kernels (and terminals) be shutdown

This doesn't seem to be the case looking at the codebase, but I may misunderstand it. https://github.com/jupyter-server/jupyter_server/blob/90f619cf11eec842a27eec25692f9d65adccf169/jupyter_server/services/shutdown.py

I don't see the KeyError issue you are mentioning, I believe I've never seen it before.

I don't see errors when shutting down the server with api/shutdown, but I see hanging processes which correspond to the kernel processes.

[kevin-bates]

This doesn't seem to be the case looking at the codebase, but I may misunderstand it. https://github.com/jupyter-server/jupyter_server/blob/90f619cf11eec842a27eec25692f9d65adccf169/jupyter_server/services/shutdown.py

Yeah, it's very subtle and I now see that this regression was introduced in #526. Previously, the exit of the io_loop triggered the _cleanup() method via the finally block. As a result, /api/shutdown calling io_loop.stop was sufficient to terminate running resources (which happened to work at that time).

I think your update via #579 is much clearer and is "moving forward" with the async method, etc.

I'll take #579 for a spin within the next couple of hours, but it looks good on the surface - thank you!

[martinRenou]

Thanks!