Local Model Support in Jupyter AI

[Sajalj98]

Problem

Currently, Jupyter AI primarily relies on cloud-based models, which can be limiting for users who have custom LLM models stored locally. Do we support for using locally available models in our file system if not we should add it.

Proposed Solution

Please add support to make use of locally available LLM models or if its already there please provide detailed steps on how to use it or direct to any available documentation.

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[JasonWeill]

Hi @Sajalj98! Thanks for contacting us. We currently support local models via GPT4All, as implemented in #209 by @krassowski . You can find documentation here: https://jupyter-ai.readthedocs.io/en/latest/users/index.html#gpt4all-usage-early-stage

If you would like to use local models via a mechanism other than GPT4All, or if you have problems with the current GPT4All implementation, please add comments to existing issues or open new issues. Thanks for your interest in Jupyter AI!

[adaaaaaa]

@JasonWeill how can i use this model locally?
https://huggingface.co/TheBloke/phi-2-GGUF/blob/main/phi-2.Q6_K.gguf