Misc. bug: --cache-reuse no longer seems to be caching prompt prefixes

[ghnp5]

This is a re-open of #14113

Name and Version

Affected:
Version at commit: b7a1746

Not affected:
Version at commit: c6a2c9e

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Problem description & steps to reproduce

I had open this bug in oobabooga/text-generation-webui: oobabooga/text-generation-webui#7060

The issue being that prompt prefixes were no longer being used in the following requests.

I confirmed that --cache-reuse 1 was being passed on, so that wasn't the issue.
After reverting to the previous version of the WebUI (which ships an older version of llama.cpp), the prompts started to be cached again.

So, this seems to point to being a bug with llama.cpp.

First Bad Commit

It looks like there may have been a commit between c6a2c9e and b7a1746 that broke --cache-reuse, or that changed its behavior.

---

I use the oobabooga/text-generation-webui project, which bundles a snapshot of llama.cpp (the commits I mentioned above),

and, from what I saw, this is the command it runs:

llama-server --model user_data/models/gemma-3-12b-it-qat-UD-Q6_K_XL.guff --ctx-size 32768 --gpu-layers 49 --batch-size 256 --port 60033 --no-webui --threads 10 --threads-batch 10 --rope-freq-scale 0.125 --rope-freq-base 1000000.0 --cache-reuse 1

The model can be downloaded from: https://huggingface.co/unsloth/gemma-3-12b-it-qat-GGUF/tree/main

I confirm --cache-reuse 1 is present in the parameters, when the cache isn't working.

I don't know how exactly a Chat Completion call is made to llama-server, but basically, if I make two chat-instruct calls, where the first 1000 characters or so are exactly the same, the prompt will be processed from byte 0, when using the affected commits I mentioned above.

I'm not providing the cache_prompt parameter, so it should default to true.

I hope this helps!

---

This is still happening when using the version at https://github.com/ggml-org/llama.cpp/tree/90083283ec254fa8d33897746dea229aee401b37

It appears that the fix from #14163 did not fix the cache issue.

Thank you.

[pwilkin]

I think you need --swa-full for that to work now (it will use more memory though).

[ggerganov]

Yes, iSWA models require --swa-full to be able to do cache reuse. The reason is that this operation require to store the full KV cache - not just the one in the sliding window.

[ghnp5]

Hey! Alright - looks like --swa-full resolves this problem.

When I was testing, the 3rd completion (which was supposed to use the cache from the previous calls) didn't use the cache, for some reason. However, all further attempts successfully used the cache, so it DOES seem to be working. Not really sure what happened on that other one, but as long as most of them use the cache, I'm happy!

Thank you so much!

[FSSRepo]

I still have a problem with GPT OSS 20B. When using the web UI, the first chat works fine — it processes the system prompt and generates the response. But on the next question, it takes a long time because it reprocesses the response it already generated before. If the previous response is very long, it takes a long time to generate a token.

[ggerganov]

@FSSRepo Could you post the llama-server command that you use?

[slaren]

I also noticed this with the 120B model. Almost every new message causes a full reprocess of the entire context, which really impacts the usability of the model. To be blunt, it makes the model unusable when keeping MoE layers in the CPU. My assumption is that it is due to differences in the tokenization and what the model generated.

[ggerganov]

Rather it's the small SWA window - just 128 tokens. Can you confirm that no extra reprocessing is observed with --swa-full?

[slaren]

I will give it a try. My understanding is that SWA makes the issue worse because any differences in the tokenization will cause the entire context to be reprocessed, rather than from the point where the difference starts. However, it should be able to reuse the context if there are no differences. Is that wrong?

[ggerganov]

This is generally true, but for reasoning models there is caveat. The thinking of response N is not included in request N+1. So regardless of tokenization differences, if the sum of thinking + generated tokens for response N is larger than n_ubatch + n_swa, we will reprocess.

With the --cache-reuse flag enabled, reprocessing will be reduced for reasoning models because we can reuse all generated tokens. But there is still a problem: if the number of thinking tokens in the response is larger than n_ubatch + n_swa we will reprocess too.

[slaren]

I didn't realize that the reasoning was being stripped from the context in this model. --swa-full greatly reduces the amount of reprocessing on each message, as expected, starting from what I assume is the start previous message, instead of reprocessing the entire context.

[ggerganov]

You can make it keep the thinking tokens from the webui:

But if I remember correctly this is not recommended and will likely confuse the model.

Note that --swa-full is one option, but you can also choose to increase n_ubatch.

If we think these are not enough, we can also add a new parameter to explicitly control the SWA cache size independently from n_ubatch - would be very easy to implement.

[slaren]

With the --cache-reuse flag enabled, reprocessing will be reduced for reasoning models because we can reuse all generated tokens. But there is still a problem: if the number of thinking tokens in the response is larger than n_ubatch + n_swa we will reprocess too.

I am skeptical of RoPE shifts because the result is not the same as if the context was regenerated, so it is hard to predict how it will affect the quality of the model output. I suppose it would be possible to keep a checkpoint of the SWA cache state at the end of every message, and it would require a lot less memory than --swa-full.

You can make it keep the thinking tokens from the webui:

I already have that option disabled in the WebUI, so if that's the case and also affects this model, then I don't think the cause of the reprocessing is likely to be the stripping of the reasoning.

Note that --swa-full is one option, but you can also choose to increase n_ubatch.

The messages tend to be rather long because they usually include a lot of code, so I don't think this would be viable, it would increase the size of the compute buffer too much.