Potential ideas for LoRA adapters

[PotatoSpudowski]

Hi,
Thank you for the repo and initial lora adapter support.

We explored a few experiments in the fastLLaMa repo.

What we did:

• Cached the lora matrix multiplication results in the convert-lora-to-ggml.py script.
  • Instead of performing the calculation during runtime,
    we cached and saved these results so when we load the adapters, it is faster.
  • We also have arguments that allow users to use the same implementation as seen in this repo.
  • We also have an argument that lets the users cache the lora adapters in fp32 or fp16 mode.
  • fp16 mode seemed to be decent and might benefit downstream applications.
• Added support for detaching lora adapters.
  • We added support to detach adapters in llama.cpp here.

Do these features seem like something that would be relevant to this repo?
If yes we would be happy to help implement these!

Happy hacking :)

[slaren]

Instead of performing the calculation during runtime, we cached and saved these results so when we load the adapters, it is faster.

I discarded this idea in the initial implementation because it increases the size of the LoRAs dramatically, which goes against the point of supporting applying LoRAs on-the-fly in the first place. I would look into optimizing the LoRA mat muls before going this path (for example #996).

fp16 mode seemed to be decent and might benefit downstream applications.

I think this may be interesting, I didn't bother too much with this because ggml doesn't support f16xf16 mat mul anyway and f16xf32 didn't seem to increase performance significantly.

We added support to detach adapters in llama.cpp here.

That's great, and we should definitely support that.

[amitsingh19975]

Instead of performing the calculation during runtime, we cached and saved these results so when we load the adapters, it is faster.

I discarded this idea in the initial implementation because it increases the size of the LoRAs dramatically, which goes against the point of supporting applying LoRAs on-the-fly in the first place. I would look into optimizing the LoRA mat muls before going this path (for example #996).

It would be great if matrix multiplication is fast enough not to cache the matrices. I did some of my thesis on optimizing BLAS operations using OpenMP, if you are interested, I can link my repo. On the other note, @PotatoSpudowski and I implemented support for both cached and uncached adapters. Furthermore, we recently added support for attaching adapters to mmaped models, and it will be merged soon.

fp16 mode seemed to be decent and might benefit downstream applications.

I think this may be interesting, I didn't bother too much with this because ggml doesn't support f16xf16 mat mul anyway and f16xf32 didn't seem to increase performance significantly.

I am more interested in quantized mat mul as it might be more performant but with less quality. We did try it, but I'm not confident with the implementation. Our current implementation was very bad, but we had a few ideas on how to fix it. However, we had to ditch it because we were not confident with the implementation or it required extra memory or storage space.

We added support to detach adapters in llama.cpp here.

That's great, and we should definitely support that.

If you need any help, feel free to ask.

[ekryski]

I would love to see the attach and detach land upstream here. Hot loading LoRAs will be incredibly useful.

[mamoesta]

I've seen a couple requests elsewhere for LoRA + base model quantizing in the repo, but saw the choice to make LoRA adapters fp16 compatible here.

I noticed HF's PeftModel class allows for base model 8bit loading + LoRA. What's the intuition behind why ggml can only do LoRA in fp16? I'd love to use 13B pre-trained LLama/Vicuna + LoRA on a 4090 but I OOM unless I can load it in 8-bit (~15 GB VRAM).

[xiaoyunwu]

I would love to see how to effectively decode with multiple loras with the same base model. that will be extremely useful.