Context Shifting

[tmsingson]

Description

About 10 days ago, KoboldCpp added a feature called Context Shifting which is supposed to greatly reduce reprocessing. Here is their official description of the feature:

NEW FEATURE: Context Shifting (A.K.A. EvenSmarterContext) - This feature utilizes KV cache shifting to automatically remove old tokens from context and add new ones without requiring any reprocessing. So long as you use no memory/fixed memory and don't use world info, you should be able to avoid almost all reprocessing between consecutive generations even at max context. This does not consume any additional context space, making it superior to SmartContext.

Any chance this gets added to Ooba as well?

Additional Context

Reddit thread: https://www.reddit.com/r/LocalLLaMA/comments/17ni4hm/koboldcpp_v148_context_shifting_massively_reduced/
llama.cpp pull: ggerganov/llama.cpp#3228
kobold.cpp 1.48.1 release: https://github.com/LostRuins/koboldcpp/releases/tag/v1.48.1

[github-actions]

This issue has been closed due to inactivity for 6 weeks. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.

[kisenera]

Is there any way to get this for Exl2?

[aarongerber]

This was closed as stale. Did it ever get implemented @oobabooga? Literally this is driving me to use KoboldCpp. As soon as you hit context limits in Oobabooga it becomes obnoxious in comparison. :/

[RichardFevrier]

Is there any way to get this for Exl2?

Wish too know too

[aarongerber]

Thanks @oobabooga you rock

[kelheor]

I remember when I started to use this project (when it just created) with pygmalion 6b a long time ago, it worked exactly like that - AI were shifted context and forgot old info without any slow down. Then at some moment, it stopped worked like that (maybe when exllama appeared). I wonder why that happened...