Adapt to run using Open-source / Local structure

[lcjr86]

Hello,
I think that the project is excellent!
Do you have plans to adapt the code to use local/open-source LLMs?

Best regards,

Luiz

[starpig1129]

Hello Luiz, Thank you very much for your suggestion about the project! Regarding plans to adapt the code for local/open-source LLMs, I have indeed considered this direction. However, due to LangChain's limited support for local models, this may require more development work. For instance, in another project of mine - a Discord bot <https://github.com/starpig1129/ai-discord-bot-PigPig> - I've already implemented local models, but it required significant adjustments to make the system work properly. Adapting this project to support local models would likely need similar modifications. If you're interested in this aspect, I'd welcome your involvement! Perhaps we could collaborate to develop a version that supports local models. Do you have any thoughts on this? I'm looking forward to your reply. Best regards, Zi Yue Chao Luiz Carlos de Jesus Junior ***@***.***> 於 2024年9月10日 週二 下午7:39寫道：

…

Hello, I think that the project is excellent! Do you have plans to adapt the code to use local/open-source LLMs? Best regards, Luiz — Reply to this email directly, view it on GitHub <#2>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AYOGMJLHV7PQH26NAQALNCDZV3LARAVCNFSM6AAAAABN6RYYFWVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZXGE3DIMZSHE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** com>

[sevapru]

Hi man, I could probably help you local models support. Could you find me anywhere by the same @ that I have in Github 🏄

[starpig1129]

Hi ，Can you send your contact information to my email?
james911129@gmail.com

[starpig1129]

@sevapru

[lcjr86]

it will be cool to have that integrated with ollama models, for example ;-)

[starpig1129]

I will probably first adjust the non-local version to be better

[jana37100]

I also would like to use it using open source models. I spent all day trying to adapt the code to Ollama which can follow the open AI structure but its a little tricky to know exactly where to edit it,

[starpig1129]

It may be necessary to stably output the model in json format
And write a class similar to ChatOpenAI

[jana37100]

In Ollama you can use similar class to OpenAI with for example:
"from ollama import Client
client = Client(
host='URL_to_hosted_model',
headers={'x-some-header': 'some-value'}
)

response = client.chat(model='llama3.1', messages=[
{
'role': 'user',
'content': 'Why is the sky blue?',
},
])
"

What I was trying to find is where best to subsititute this with what you have in the code, just with main.py, and the main. notebook?

[starpig1129]

Maybe you can start modifying it from here
https://github.com/starpig1129/AI-Data-Analysis-MultiAgent/blob/main/core/language_models.py