Jupyterhub kernel access no longer works for users with certain special characters in their username #53

quantum-byte · 2024-03-27T11:48:31Z

With newer vscode-jupyter extension versions than v2023.9.1102792234, it is no longer possible to access / use Jupyterhub kernels for Jupyterhub users with special characters in their username like user (test).

Its the same issue somebody raised in issue 15055 as far as i can see but this one was closed.

This issue is also not in the separate ms-toolsai.jupyter-hub extension. The authentication works fine but the subsequent kernel iteration through vscode-jupyter breaks.

Environment data

VS Code version: Version: 1.87.2
Jupyter Extension version (available under the Extensions sidebar): everything greater than v2023.9.1102792234 e.g. v2024.2.0
Python Extension version (available under the Extensions sidebar): v2024.2.1
OS (Windows | Mac | Linux distro) and version: Linux Ubuntu
Python and/or Anaconda version: 3.10
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): conda/mamba
Jupyter server running: Tested on a Local and Remote Jupyterhub instance (docker setup).
One for the bug potential import config detail is that we use the separate proxy setup.

Expected behaviour

The specific Jupyterhub user can login via the web ui and their server instance is started.
They then can use the Jupyterhub Server URL, username and generated token to setup a new Jupyterhub connection to access and use the kernels there in VSCode. Finally they can use one of the shown kernels to execute a example Notebook.

This also works if one uses vscode-jupyter extension versions v2023.9.1102792234.

Actual behaviour

The specific Jupyterhub user can login via the web ui and their server instance is started.
They then can use the Jupyterhub Server URL, username and generated token to setup a new Jupyterhub connection to access and use the kernels there in VSCode.

Get an error Cant connect to Remoteserver after the Jupyterhub connection setup is complete and i cant see any kernels from the remote server to use. You can find the complete Extension Output at the bottom.

The relevant line from the output is http://localhost:8000/user/user%20(test)/api/kernels?1711536811205. What works and what the Jupyterhub expects is http://localhost:8000/user/user%20%28test%29/api/kernels?1711536811205.

I am unsure if the error is only due to our use of the Jupyterhub separate proxy deployment or if the Jupyterhub internal proxy also has this issue, but it is very likely.

The fact is that jupyterhub encodes every special character except ~ and @ for the username and expects the username to be encoded in request paths like that. This can partially be seen here and here. For some background behind this behaviour i found also this issue which talks about it.

Basically what happens then is, that the Jupyterhub proxy knows to redirect a request for http://localhost:8000/user/user%20%28test%29/ but not for http://localhost:8000/user/user%20(test)/. That is why we see the 404 error in the log output.

My best guess is you broke this with microsoft/vscode-jupyter@4b47894 by no longer encoding some characters in the whole base url which also contains the username. But i think it was also not fully consistent with how Jupyterhub expects everything to be encoded before.

Also as far as i could see the vscode-jupyterhub extension is not affected by this issue because they retrieve the user info server property from the Jupyterhub server and build the server user instance url by appending it to the base url as can be seen here (At least in the happy path where the Jupyter server returns the server property for the user. The fallback implementation looks buggy). The Jupyterhub user info server property contains the correctly encoded username.

Steps to reproduce:

[NOTE: Self-contained, minimal reproducing code samples are extremely helpful and will expedite addressing your issue]

Setup or use a local or remote Jupyterhub instance. Login in and start the server with a user called user (test). This is quite easy to simulate with the DummyAuthenticator authenticator module which allows any kind of username to be tested.
Setup a API Tokens for authentication on the Jupyterhub instance
Use the url, user name and token and create a new remote Jupyterhub connection
1. Select existing Jupyterhub-Server
2. Select enter URL for existing Server
3. Enter URL for Jupyterhub server. In my local example http://localhost:8000
4. Enter Username for Jupyterhub server instance. In my local example user (test)
5. Enter previously generated Jupyterhub server instance token as Password.
6. Enter a name under which Vscode saves the Jupyterhub server. In my local example JupyterHub (test)
7. Get the error Cant connect to Remoteserver. See also the image (its in german though)
8. The Jupyterhub connection details are saved but every time i try to access it via VSCode i get the above error.

Logs

Output for Jupyter in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Jupyter)

11:53:31.116 [info] Start refreshing Kernel Picker (1711536811116)
11:53:31.211 [error] Error fetching kernels from http://localhost:8000/user/user (test)/ (JupyterHub (test)): [r [Error]: Invalid response: 404 Not Found
	at r.create (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/node_modules/@jupyterlab/services.js:11:1067)
	at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
	at async Object.Rl [as listRunning] (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/node_modules/@jupyterlab/services.js:11:4507)
	at async r.getRunningKernels (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/extension.node.js:198:12975)
	at async Promise.all (index 0)
	at async sD.listKernelsFromConnection (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/extension.node.js:308:19197)
	at async ~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/extension.node.js:308:17485] {
  response: p [Response] {
    size: 0,
    timeout: 0,
    [Symbol(Body internals)]: { body: [PassThrough], disturbed: true, error: null },
    [Symbol(Response internals)]: {
      url: 'http://localhost:8000/user/user%20(test)/api/kernels?1711536811205',
      status: 404,
      statusText: 'Not Found',
      headers: [p [Headers]],
      counter: 0
    }
  },
  traceback: ''
}
11:53:31.213 [error] UniversalRemoteKernelFinder: Failed to get kernels without cache [r [Error]: Invalid response: 404 Not Found
	at r.create (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/node_modules/@jupyterlab/services.js:11:1067)
	at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
	at async Object.Rl [as listRunning] (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/node_modules/@jupyterlab/services.js:11:4507)
	at async r.getRunningKernels (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/extension.node.js:198:12975)
	at async Promise.all (index 0)
	at async sD.listKernelsFromConnection (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/extension.node.js:308:19197)
	at async ~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/extension.node.js:308:17485] {
  response: p [Response] {
    size: 0,
    timeout: 0,
    [Symbol(Body internals)]: { body: [PassThrough], disturbed: true, error: null },
    [Symbol(Response internals)]: {
      url: 'http://localhost:8000/user/user%20(test)/api/kernels?1711536811205',
      status: 404,
      statusText: 'Not Found',
      headers: [p [Headers]],
      counter: 0
    }
  },
  traceback: '',
  vslsStack: [
    CallSite {},
    CallSite {},
    CallSite {},
    CallSite {},
    CallSite {},
    CallSite {},
    CallSite {}
  ]
}
11:53:31.214 [error] Jupyter Lab Helper:getKernelSpecs failure:  [Error: Poll (@jupyterlab/services:KernelSpecManager#specs) is disposed.
	at v.dispose (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/node_modules/@jupyterlab/services.js:11:10935)
	at ni.dispose (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/node_modules/@jupyterlab/services.js:11:29299)
	at r.dispose (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/extension.node.js:198:12579)
	at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
	at async Promise.all (index 0)
	at async sD.listKernelsFromConnection (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/extension.node.js:308:20060)
	at async ~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/extension.node.js:308:17485]
11:53:31.237 [info] End refreshing Kernel Picker (1711536811116)

The text was updated successfully, but these errors were encountered:

DonJayamanne · 2024-03-27T12:39:40Z

Thank you very much for the detailed analysis, and providing the great sample for being able to easily repro this.
I will take a look at this soon, and will try to get this fixed for the next release (unfortunately we will be shipping a new version next week & it is too late to get this fixed for that).

DonJayamanne · 2024-03-27T22:53:58Z

@quantum-byte
Please can you install this version of Jupyter Hub extension, & let me know if this works for you.
Tested at my end and works (again thanks for the great steps)

ms-toolsai-jupyter-hub-insiders.vsix.zip

quantum-byte · 2024-03-28T11:08:12Z

Hi @DonJayamanne , thanks for looking into it so quickly!

Please can you install this version of Jupyter Hub extension, & let me know if this works for you.
Tested at my end and works (again thanks for the great steps)

I tested it and i can execute my Test Notebook successfully as far as i can see. But i still see a few Errors in the vscode-jupyter extension output regarding some extension.js not being able to be downloaded (see below). Not sure if this is related or something else entirely.

As an example for one of the urls if i use the adjusted http://localhost:8000/user/user%20%28test%29/nbextensions/jupyterlab-plotly/extension.js i get the following successful response:

// Entry point for the notebook bundle containing custom model definitions.
//
define(function() {
    "use strict";

    window['requirejs'].config({
        map: {
            '*': {
                'jupyterlab-plotly': 'nbextensions/jupyterlab-plotly/index',
            },
        }
    });
    // Export the required load_ipython_extension function
    return {
        load_ipython_extension : function() {}
    };
});

I am also quite surprised, that the bugfix needed to be done in the jupyterhub extension. I guess i did not understand your sourcecode well enough as i though :).

Output for Jupyter in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Jupyter)

Visual Studio Code (1.87.2, undefined, desktop)
Jupyter Extension Version: 2024.2.0.
Python Extension Version: 2024.2.1.
Pylance Extension Version: 2024.3.2.
Platform: linux (x64).
Multiple Workspace folders opened <**********>
10:56:41.124 [info] Start refreshing Kernel Picker (1711619801124)
10:56:41.138 [info] Using Pylance
10:56:42.828 [info] Start refreshing Interpreter Kernel Picker
10:56:45.913 [info] Process Execution: ~/.mambaforge/envs/testenv/bin/python -m pip list
10:56:46.041 [info] No interpreter for Pylance for Notebook URI "<******|.ipynb"
10:56:51.631 [info] End refreshing Kernel Picker (1711619801124)
10:56:55.443 [info] Start refreshing Kernel Picker (1711619815443)
10:56:55.682 [info] End refreshing Kernel Picker (1711619815443)
10:56:59.198 [info] Handle Execution of Cells 5,8,10,14,15,20,22,30 for <******|.ipynb
10:56:59.200 [info] Starting Kernel startUsingRemoteKernelSpec, 7e801c78ddcea37e824adca34211c78ef7f7527d1106c3d51cad9b3a0be0e7a5..testenv./opt/conda/envs/testenv/python./.-m#ipykernel_launcher  for '<******|.ipynb' (disableUI=false)
10:56:59.256 [info] http://localhost:8000/user/user (test)/: Kernel gestartet: 4a9cd58b-cfbd-48e5-81d6-627bc3fbca75
10:56:59.782 [info] Started session for kernel startUsingRemoteKernelSpec:7e801c78ddcea37e824adca34211c78ef7f7527d1106c3d51cad9b3a0be0e7a5..testenv./opt/conda/envs/testenv/python./.-m#ipykernel_launcher
10:56:59.900 [info] Registering Kernel Completion Provider from kernel Python 3 (testenv) for language python
10:56:59.900 [info] Start refreshing Kernel Picker (1711619819900)
10:57:00.035 [info] Kernel acknowledged execution of cell 5 @ 1711619820034
10:57:00.248 [info] End refreshing Kernel Picker (1711619819900)
10:57:01.504 [info] End cell 5 execution after 1.47s, completed @ 1711619821504, started @ 1711619820034
10:57:01.535 [info] Kernel acknowledged execution of cell 8 @ 1711619821535
10:57:01.538 [error] Error downloading from http://localhost:8000/user/user (test)/nbextensions/jupyter-js-widgets/extension.js: Not Found
10:57:01.538 [warn] Failed to fetch script sources [TypeError: Cannot read properties of undefined (reading 'replace')
	at Hc.ourFetch [as fetchImplementation] (~/.vscode/extensions/ms-toolsai.jupyter-hub-2024.4.100/dist/extension.node.js:68102:32)
	at Hc.downloadFile (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/extension.node.js:265:13637)
	at UT.getWidgetScriptSourceUsingFetch (~/.vscode/extensions/ms-toolsai.jupyter-2024.2.0-linux-x64/dist/extension.node.js:341:86140)
	at process.processTicksAndRejections (node:internal/process/task_queues:95:5)]
10:57:01.541 [error] Error downloading from http://localhost:8000/user/user (test)/nbextensions/jupyterlab-plotly/extension.js: Not Found
10:57:01.777 [info] End cell 8 execution after 0.242s, completed @ 1711619821777, started @ 1711619821535
10:57:01.788 [info] Kernel acknowledged execution of cell 10 @ 1711619821788
10:57:01.814 [info] End cell 10 execution after 0.026s, completed @ 1711619821814, started @ 1711619821788
10:57:01.821 [info] Kernel acknowledged execution of cell 14 @ 1711619821821
10:57:01.879 [info] End cell 14 execution after 0.058s, completed @ 1711619821879, started @ 1711619821821
10:57:01.886 [info] Kernel acknowledged execution of cell 15 @ 1711619821886
10:57:09.512 [info] End cell 15 execution after 7.626s, completed @ 1711619829512, started @ 1711619821886
10:57:09.518 [info] Kernel acknowledged execution of cell 20 @ 1711619829518
10:57:09.748 [info] End cell 20 execution after 0.23s, completed @ 1711619829748, started @ 1711619829518
10:57:09.753 [info] Kernel acknowledged execution of cell 22 @ 1711619829753
10:57:12.043 [info] End cell 22 execution after 2.29s, completed @ 1711619832043, started @ 1711619829753
10:57:12.049 [info] Kernel acknowledged execution of cell 30 @ 1711619832049
10:57:12.558 [info] End cell 30 execution after 0.509s, completed @ 1711619832558, started @ 1711619832049

Regarding

(unfortunately we will be shipping a new version next week & it is too late to get this fixed for that)

That is fine. For now we can tell the small amount of affected users to downgrade to the older extension version.

DonJayamanne · 2024-04-01T21:04:44Z

that the bugfix needed to be done in the jupyterhub extension. I guess i did not understand your sourcecode well enough as i though :).

Your assessment was accurate, however I didn't want to modify the Jupyter extension to get around the issue with the Jupyter Hub problem.

Here's the solution that keeps this fix in the Jupyter Hub extension #52
This way Url encoding thats specific to Hub stays there, and other Urls are treated differently.

not being able to be downloaded (see below). Not sure if this is related or something else entirely.

You can ignore this logged error, its a noop, it should work, please test and let me know if plotly widgets work/or not.
From what I can tell it should work and the error thats logged is a false positive (will modify to not log this)

quantum-byte · 2024-04-02T10:57:09Z

that the bugfix needed to be done in the jupyterhub extension. I guess i did not understand your sourcecode well enough as i though :).

Your assessment was accurate, however I didn't want to modify the Jupyter extension to get around the issue with the Jupyter Hub problem.

Here's the solution that keeps this fix in the Jupyter Hub extension #52 This way Url encoding thats specific to Hub stays there, and other Urls are treated differently.

Right, that totally makes sense! Fix it where the responsibility lies.

not being able to be downloaded (see below). Not sure if this is related or something else entirely.

You can ignore this logged error, its a noop, it should work, please test and let me know if plotly widgets work/or not.
From what I can tell it should work and the error thats logged is a false positive (will modify to not log this)

Its not quite a noop as far as i can tell. In the end it works in both cases since the fallback is to download the extension code from the cdn.

But it still is a inconsistent behaviour since for "normal" usernames without special characters the extension code is downloaded from the Jupyterhub instance itself instead of the cdn. I opened a separate Ticket for it: #55

quantum-byte added the bug Issue identified by VS Code Team member as probable bug label Mar 27, 2024

quantum-byte mentioned this issue Mar 27, 2024

Using an email address as a username fails to get a list of kernels #41

Closed

DonJayamanne self-assigned this Mar 27, 2024

quantum-byte mentioned this issue Mar 27, 2024

Can't connect to jupyterhub #45

Open

DonJayamanne added this to the April 2024 milestone Mar 27, 2024

DonJayamanne transferred this issue from microsoft/vscode-jupyter Apr 1, 2024

This was referenced Apr 1, 2024

Handle special characters in user names #52

Merged

Do not log errors unnecessarily microsoft/vscode-jupyter#15456

Merged

DonJayamanne closed this as completed in #52 Apr 2, 2024

quantum-byte mentioned this issue Apr 2, 2024

Jupyterhub kernel access with extensions for users with certain special characters in their username fails to access extension.js code from Jupyterhub #55

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jupyterhub kernel access no longer works for users with certain special characters in their username #53

Jupyterhub kernel access no longer works for users with certain special characters in their username #53

quantum-byte commented Mar 27, 2024

DonJayamanne commented Mar 27, 2024

DonJayamanne commented Mar 27, 2024

quantum-byte commented Mar 28, 2024

DonJayamanne commented Apr 1, 2024 •

edited

Loading

quantum-byte commented Apr 2, 2024

Jupyterhub kernel access no longer works for users with certain special characters in their username #53

Jupyterhub kernel access no longer works for users with certain special characters in their username #53

Comments

quantum-byte commented Mar 27, 2024

Environment data

Expected behaviour

Actual behaviour

Steps to reproduce:

Logs

DonJayamanne commented Mar 27, 2024

DonJayamanne commented Mar 27, 2024

quantum-byte commented Mar 28, 2024

DonJayamanne commented Apr 1, 2024 • edited Loading

quantum-byte commented Apr 2, 2024

DonJayamanne commented Apr 1, 2024 •

edited

Loading