Inconsistent experience with respect to non-installed third-party packages

[krassowski]

Description

On a fresh install there are no Open AI models listed any more; this is similar for Anthropic. This tripped many users:

• 🐛 chatgpt not available anymore in 2.16.0 : _"Cannot determine model provider from model ID openai-chatgpt."_ #803
• OpenAI language models do not show in the language selection dropdown #721
• Example from readme not working #746
• There seems to be a problem with the chat backend, please look at the JupyterLab server logs #508 (comment)

Yet, the Hugging Face Hub is listed. When user selects it and attempts to use it it throws an error:

pydantic.error_wrappers.ValidationError: 1 validation error for HfHubProvider __root__ Could not import huggingface_hub python package. Please install it with `pip install huggingface_hub`. (type=value_error)

Expected behavior

• Either all models that jupyter-ai supports out of the box are shown or no models are shown by default or none of them are shown by default.
• When a package is needed, the model is greyed out and user is prompted to install it. Ideally without restarting Jupyter server.

Context

• Related to Chat panel fails without selected provider dependencies installed #680

[krassowski]

Can I get some more eyes on this one? I saw this trip dozens of users now:

• OpenAI models not listed #1031
• Support for OpenAI's GPT-4o-mini disappeared in jupyter-ai-2.23.0 settings  #999
• Cannot see Azure provider models #983
• Cannot determine model provider from model ID openai-chat:gpt-3.5-turbo. #973
• Bedrock models are no longer available (version 2.20.0) #921
• OpenAI language models do not show in the language selection dropdown #721
• No 'OpenAI' model in model selection. #719
• There seems to be a problem with the chat backend, please look at the JupyterLab server logs #508 (comment)
• https://discourse.jupyter.org/t/jupyter-ai-no-openai-choice-in-completion-model-dropdown/29130

I would suggest that all models from built-in providers are always listed (which can be configurable), but those which require installing a package (which is missing) are indicated with an icon. If such a model is selected, the pip install/conda install (if available) command should be shown for the user to copy and run.

[krassowski]

Any thoughts or design (counter)proposals @dlqqq @srdas?

While it is technically feasible to implement hot-installation without restarting the jupyter-server that would only work if the user is running in an environment which is not read-only (so requires a fallback path), hence for the first iteration I would suggest to only display information (ideally a ready-to-paste command) on what is needed.

[dlqqq]

@krassowski Yes, I agree that this has been a major pain point for our users, and deserves more attention from us contributors. This was one of the things I wanted to fix right after Jupyter AI was presented at JupyterCon in May 2023, but other feature requests and fixes have repeatedly taken priority over this issue. I would like to work on this soon, but since resolving this issue would require a large time commitment, I need to get approval to work on this from my management chain.

Let me offer some feedback here:

I would suggest that all models from built-in providers are always listed (which can be configurable), but those which require installing a package (which is missing) are indicated with an icon. If such a model is selected, the pip install/conda install (if available) command should be shown for the user to copy and run.

hence for the first iteration I would suggest to only display information (ideally a ready-to-paste command) on what is needed.

One concern I have is that it's unclear how the server extension would be able to know what models are not available due to the lack of a dependency package. We use the Entry Points API to load modules, which contain providers, which list available models under the models class attribute. However, if an optional dependency is not installed, the modules cannot be loaded. If the modules cannot be loaded, then we can't read the class that defines the list of models.

This is a fundamental issue with our concept of "providers". Each provider needs to subclass a LangChain LLM or ChatModel class, which generally requires an optional dependency. If the optional dependency is not installed, we don't know what models or providers are not available. I don't immediately see an obvious way of working around this in a minor release.

Here are the counter-proposals I've considered thus far (all of which would likely require a major release):

1. Discard the concept of providers & build a new class that can declare its own metadata (e.g. the list of model IDs supported) without requiring an optional dependency to be loaded.
   • I have given considerable thought to replacing the notion of providers in this project. A provider is too many things: it declares metadata, it is a subclass of a LangChain class, and it can declare its own custom persona. These concerns should be separated into dedicated classes that can be changed independently, instead of all being implemented together in an ambiguously-defined "provider" class.
   • For example, we could replace BaseProvider with a new class BaseModelSet, which doesn't inherit from any optional dependency. This BaseModelSet class would need to define a new class method like get_model_instance(), which returns an instance of a LangChain LLM or ChatModel class. The key here is that the get_model_instance() method can dynamically import the optional dependency. Thus, this potentially allows us to list all available LLMs without needing to install all optional dependencies.
2. Perform all operations in a separate Python environment (via venv or conda).
   • This would be fantastic because we would no longer need to worry about dependency conflicts or even Python version compatibility. Since everything is done in an environment we manage, we can ensure a functional experience out-of-the-box regardless of the user's current environment. This would probably also allow us to support dynamic installation (i.e. not require a restart after installing an optional dependency).
   • However, I have not done much research into exactly what this would require or if it is even feasible. My initial impression is that this would require extensive inter-process communication, which may be too difficult to implement & maintain.
   • Given this, option 1) is probably more realistically achievable than option 2) here.

Let me know what you think!

[ellisonbg]

Yeah, I keep running into this issues on new installs as well.

[dlqqq]

@krassowski Actually, there may be a way to support this in a minor release. @ellisonbg suggested that we could potentially show still show providers that are missing optional dependencies, without showing the full list of model IDs. These providers would still be listed in the model dropdown in the Settings UI, but when selected, they would prompt the user to install the optional dependency & restart the JupyterLab server, and not allow changes to be saved.

[krassowski]

One concern I have is that it's unclear how the server extension would be able to know what models are not available due to the lack of a dependency package. We use the Entry Points API to load modules, which contain providers, which list available models under the models class attribute. However, if an optional dependency is not installed, the modules cannot be loaded. If the modules cannot be loaded, then we can't read the class that defines the list of models.

One way to go about it would be using conditional imports in provider modules (if import fails we could use a shim which still allows to load the provider, but also sets some special attribute to distinguish it).