Mistral AI integration

[mathieubrunpicard]

HostedGPT currently includes Claude & OpenAI but could we also have a Mistral AI integration ?

I see that there are two lib that could be used in Ruby :

• https://github.com/fsndzomga/mistral_rb
• https://github.com/gbaptista/mistral-ai
• https://github.com/wilsonsilva/mistral

the latter might seem the better option has it mirrors the mistral python lib (cf. https://wilsonsilva.com/mistral-ruby-gem/)

[krschacht]

Hi @mathieubrunpicard converted to a discussion just to help me stay organized. Thanks for raising this question. I know one other contributor also wants the project to run local models and he's doing some investigating of it. cc: @stephan-buckmaster , not sure how far along you are.

Do you chat with Mistral currently? I haven't tried it yet. I just keep checking the public evals and it seems like Llama3 is the strongest of the open source models: https://chat.lmsys.org/?leaderboard

I do think it would be great to support Llama3 and probably Mistral too. The challenge isn't with the ruby side, it's really the question of where to host the model. Are we using Grock and pointing to their web API? Or do we ask a user to run it locally? And if you run it locally, do you know if it conforms to the OpenAI API spec?

If you're running it locally, what are you using? When I've run Llama I use the llamafile format: https://github.com/Mozilla-Ocho/llamafile

Thinking aloud, maybe the task is this:

• First we get PR Create new & manage Assistants #334 merged in which lets people manage Assistants
• Then, maybe we add a section to Settings that lets you create new Language Models. When you create one, I think all you would need is to specify URL, port, and some credentials. We then just assume it conforms to the OpenAI API spec?
• Then you could create an assistant that used this custom LanguageModel that you added.

[mathieubrunpicard]

No worries @krschacht.

I'm using Hugging Chat with Mixtral-8x7B-Instruct-v0.1 (amongst others)
I haven't compared it directly to Llama3 so I can't speak as the differences between the two.

Personnaly I don't run it locally.
That being said, I think that what would make the most sense would be to follow the current implementation pattern in the app (i.e. plug & play with an API key). This would mean a Groq integration (as Groq comes with both Llama3 & Mixtral-8x7b).

[krschacht]

@mathieubrunpicard I've not heard of Hugging Chat, that looks nice.

It looks like Groq uses the OpenAI API spec with only some minor differences, so that shouldn't be too much work.

If we were to add support for Mistral, is there something that you would like this Hosted GPT app to do that Hugging Chat isn't doing for you?

[krschacht]

Quick update on this, @mathieubrunpicard , we are just about done adding support for any LLMs provided it conforms to the OpenAI or Anthropic API, which most do. And if we needed to add another adapter it wouldn’t be too much work to do that. @stephan-buckmaster is working on PR #389 which is the second half of this project. We already got the first part which is the ability to create / edit your assistants.