Handle non-existing kernels

[jtpio]

Fixes #259. It should also provide a simple solution for #267.

This PR makes use of the KernelSpecManager to first check that the requested kernel name (from the notebook metadata) is available. If it isn't then it will default to the default kernel name.

I also got hit by that recently while using xeus-python and decided to give it a shot.

[jtpio]

Another approach would be to make use of the language field from the kernelspec metadata to find a spec that matches the same language.

[SylvainCorlay]

What does the classic notebook do? It appears to be quite liberal with kernelspecs.

[jtpio]

It looks like most of the logic for dealing with missing kernel names in the classic notebook is actually handled by the frontend.

This is what it appears to be doing:

1. The frontend fetches the list of kernelspecs. When a spec is not found it logs it in the dev tools console: https://github.com/jupyter/notebook/blob/master/notebook/static/notebook/js/kernelselector.js#L226-L227

2. It then defaults to the first match using the language field, and emits the spec_changed.Kernel event: https://github.com/jupyter/notebook/blob/e498de6775b3de57f5ff827e562c179b17893064/notebook/static/notebook/js/kernelselector.js#L249

3. The spec_changed.Kernel event is then consumed here: https://github.com/jupyter/notebook/blob/master/notebook/static/notebook/js/notebook.js#L345-L357

4. A new session is started using the name from 2.: https://github.com/jupyter/notebook/blob/e498de6775b3de57f5ff827e562c179b17893064/notebook/static/notebook/js/notebook.js#L356

I might have missed some details but this is basically the outcome of stepping through the code in the dev tools.

[SylvainCorlay]

Thanks for looking into this. What do you think is the cleanest thing to do?

It seems that jupyterlab pops up a dialog to select a kernel.

Or such a logic could be in the backend?

[jtpio]

Finding a match based on the language field would certainly help with many of the common issues related to:

• having "name": "python2" in the notebook metadata
• conda named kernels: "name": "conda-env-foo-py"

And in some way it follows the logic from the classic notebook.

For the case where there is no match for a given language, for example trying to execute a C++ notebook on a Voila setup that only has Python kernels installed, this would still return a 500.

[jtpio]

The last commits implement finding a spec matching the language field from the metadata, as well as sorting the matches by display name (cf notebook implementation).

If there is no language in the metadata, it defaults to python.

[maartenbreddels]

I like this approach, my remaining question is: should we do this in voila or nbconvert.

I'm ok with getting this into voila first, then see if we can upstream this.

[maartenbreddels]

I wonder if we have 0 matches, if we can have a custom template with a proper error msg and http error code: https://en.wikipedia.org/wiki/List_of_HTTP_status_codes

If there are 1 or more matches, I think we should log a warning, so users are aware of it (I can imagine someone expecting it to run with a conda-env-kernel, which cannot be found, leading to surprises).

[jtpio]

Agreed, a warning would definitely be useful to let the users know the kernel being used is not the one they expect.

Having a custom template with a proper error message is also a good idea, but maybe for a follow-up PR?

[maartenbreddels]

Why is yield tornado.gen.maybe_future needed? The current implementation gives a plain dict, or can other implementations return a future?

[jtpio]

The kernel spec manager could be a GatewayKernelSpecManager, in which case the maybe_future would be needed: https://github.com/jupyter/notebook/blob/e498de6775b3de57f5ff827e562c179b17893064/notebook/gateway/managers.py#L521

[jtpio]

Not sure it was discussed before, but do we need to keep supporting Python 2?

[maartenbreddels]

Not really, but we have it now, and it doesn't lead yet to any major hurdles. I'd say the incentive to keep supporting is decaying exponentially though.

did it myself

[jtpio]

Right, the hurdles are quite minimal for now.

[jtpio]

@maartenbreddels this looks good to me.

Returning a 500 when there is no matching kernel is definitely useful.
The other case (showing a warning when 1+ matches found) could be added in a different PR.