Revisit extension loading mechanism

[Zsailer]

Background

I've been doing a deep dive into the extension loading design and finding it lacks convention.

The most documented way to create an extension (and thus, probably the most used) is to write a load_jupyter_server_extension function.

Right now, the main ServerApp loads any modules listed in it's jpserver_extensions trait and looks for a load_jupyter_server_function. If it's not there, it skips the extension and throws a warning.

But there is also a toggling mechanism for enabling/disabling extensions. This requires an extra method to be defined in your extensions that makes them discoverable, namely _jupyter_server_extension_paths. This function returns a list of dictionaries which represents metadata for all extensions in an extension module (yes, you can define multiple extensions in a single package). Typically, these dictionaries has a module key:value pair that describes where the load_jupyter_server_function is defined for that extension. i.e.

def _jupyter_server_extension_paths():
     return [{
         'module': my_extension
     }]

There are various issues that arise here. Right now, extension loading only uses the load_jupyter_server_function and assumes its at the root of an extension module. On the other hand, the extension toggler only uses the _jupyter_server_extension_paths function to verify the existence of an extension (or multiple extensions). Having different, inconsistent entrypoints to extensions is extremely confusing.

Further, if _jupyter_server_extension_paths defines a different path to the load_jupyter_server_extension function, it cannot be loaded by the server. What's the purpose of the first function then?

I propose that we merge these two approaches and heavily document it.

Proposed change

Make ServerApp and the extension toggler require/use the _jupyter_server_extension_paths function to discover server extensions. If this function is not defined, the extension cannot be loaded (and warnings/errors will be raised).

Right now, if an extension author does not define a _jupyter_server_extension_paths, the extension can still discovered by the server even though the toggle mechanism throws "extension not found" error. I want to explicitly require extensions to define this function.

[kevin-bates]

@Zsailer - thank you for spending the time and effort on this! I think it's important that this be well-defined and I agree that removing ambiguity is the right approach.

Do you know why _jupyter_server_extension_paths() is underscore-prefixed? My python naivete may be shining bright, but I thought that convention was reserved for private methods and this particular method is about as public as they come. I'm hoping we could perhaps deprecate that method name and document the use of jupyter_server_extension_paths(). If the provided module doesn't have the new method, then we look for the original (perhaps logging a deprecation warning at that time).

[Zsailer]

Do you know why _jupyter_server_extension_paths() is underscore-prefixed?

Great question. I think this function is private because the author of an extension would likely want this function invisible to their users. Yes, developers have to define this function when developing their extension, but their users should never call this function.

Put another way, this function is not a public API we (Jupyter server) provide; rather, it's a "protocol" we ask extension developers to follow to link into Jupyter Server. Users never see this machinery as a public API.

Does that seem reasonable?

[kevin-bates]

Yes, that makes sense. Then, by extension, shouldn't load_jupyter_server_extension be prefixed? Still not clear on why the two would be handled differently in this aspect.

[Zsailer]

Yes. I think they should both be prefixed with _. 😎 This is another inconsistency of this mechanism. We may have to support the non-prefixed function name for a bit to maintain some backwards compatibility unfortunately.

[Zsailer]

Solved in #180!