Jupyter Server imports extensions even when they are disabled, slowing down startup

[yuvipanda]

Description

jupyter server will do an import of the python package for all extensions, even if they are explicitly disabled. This can cause a big performance penalty, as well as cause code to execute that perhaps shouldn't be.

Reproduce

1. Install jupyter_server on a fresh virtualenv / conda env
2. Install the panel package, as that provides a server extension that takes a while to start
3. Run jupyter server extension list to make sure that panel.io.jupyter_server_extension is enabled
4. Disable it with jupyter server extension disable panel.io.jupyter_server_extension, and verify with jupyter server extension list again
5. Start jupyter server

Expected behavior

The panel.io.jupyter_serverextension package is not imported, and the line Package panel.io.jupyter_server_extension took 0.4643s to import is not printed by jupyter-server on startup. The extension is disabled, and should not be imported.

Instead, the line is printed as the python package is imported anyway.

Context

This came up in the context of debugging 2i2c-org/infrastructure#2047, where I was trying to figure out why jupyter server startup was so slow jupyterhub was timing out at 30s for the python process to startup after the container has started. I was testing by assuming that disabling an extension wouldn't import it, both as it is the least surprising behavior but also because that is how notebook server behaves. Eventually strace output led me to realize that disabling extenions doesn't stop them from being imported.

While #1176 and #1171
make it easier to see which packages are adding startup latency, it is still impossible to disable an extension temporarily to remove the latency issue. The only way to reduce latency from panel's serverextension is to entirely uninstall panel, which breaks other functionality too.

Given that enable / disable status can be determined without having to import the python package, no import should ideally be done if the extension itself is disabled.

[yuvipanda]

Thanks a lot @kevin-bates, @minrk and @blink1073!