Are there any plans to support GPU offloading with LORAs?

[l3utterfly]

Are there any plans to support this?

Reading some of the past issues, seems the main thing blocking is that CUDA uses f32 whilst LORA uses f16 tensors.

Is that still the case?

I can give a shot at implementing this if someone can give me a rough rundown on all the hurdles.

[abc-nix]

Hi.

I think @JohannesGaessler has already started to work on it. At least they published #1970 pull request with a rough way to make it work.

[JohannesGaessler]

Yes, I have already started working on it due to the advent of SuperHOT and will try to finalize it soon.

[JohannesGaessler]

I fixed up the PR. Also I should clarify: the PR only enables CUDA acceleration for f16 models. I previously misunderstood how ggml LoRAs are applied. What needs to be done is to modify the weights with the LoRA which is complicated by the fact that this is done after they're already in VRAM where regular ggml operations can't reach them.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.