Quantized BakLLaVA (Mistral + LLaVA 1.5) GGUF + 10 lines of Python

[paschembri]

Congrats to everyone who worked on the LLaVA integration (both on llama.cpp and llama-cpp-python).

I converted and quantized BakLLaVA model (weights here: https://huggingface.co/advanced-stack/bakllava-mistral-v1-gguf) and wrote a quick tutorial there: https://advanced-stack.com/resources/multi-modalities-inference-using-mistral-ai-llava-bakllava-and-llama-cpp.html

[cmp-nct]

Interesting
I just played around a bit with Bakllava and compared it to llava 1.5 7B and 13B
I found Bakllava to be very weak in following the actual prompt, especially trying to make it respond long or short is ignored no matter how I tried it. With llava 1.5 7B (and 13B) any of the methods I tried on Bakllava were successful, the 1.5 model responds the way requested.

Maybe Bakllava 2 will be better in that regard

[paschembri]

I didn't perform evaluations but someone on the mistral discord pointed that the Dolphin 2.1 model was giving great results. So many models to choose from ...

[cmp-nct]

Though that means to retrain the entire projector, no ?

Regarding Bakllava: as far as I can tell it's using the normal Mistral foundation model, not an instruct or chat model.
I believe that's the cause of the problem, their base model has never learned to follow instructions, so it can only generate a description and ignores the prompt mostly.
The training process of llava is quite simple, not enough to teach the model following prompts if it doesn't already have that capability.

Dolphin would have certainly been a better choice!