[Feature request] Hybrid model cache: add --checkpoint-every-nb

[aoleg]

Mainline llama.cpp added ggml-org#20087

I suggest adding a UI feature for hybrid models (such as Nemotron-H, Qwen 3.5, Jamba2-Mini) in the Context section to account for this feature (notably, context shifting is not supposed to work with RNN models, so these should likely be mutually exclusive).

Essentially, this creates cache checkpoints after processing every n batches during prompt processing. Why: it is really helpful when using apps such as SillyTavern that can insert an "anchoring" prompt N messages above the user's prompt (this is common practice to make the model adhere to system prompt guidelines). Without this feature, RNN models invalidate the whole cache, and the entire prompt must be reprocessed. With this feature, however, only the n batches of prompt caching are invalidated and must be reprocessed, which speeds up the usage significantly.

[LostRuins]

Yes we have that, its enabled by default in RNN models now if you select context shift, it will attempt to do this instead,

you can control the cache slots with --smartcache N. it will intelligently set and clear the checkpoints.

[aoleg]

No, this is something else. The PR I mentioned was only merged 5 days ago, please check it out: ggml-org#20087

What you are referring to is SmartCache, which (as far as I understand it) creates a checkpoint in RAM after processing the whole prompt. 20087 provides a better/more complete fix for the issue.

Basically, PR ggml-org#20087 adds intermediate prefix checkpoints calculated automatically during prefill, at every n batches, so hybrid/recurrent models are more likely to successfully resume from an earlier saved prefix. While it stores several snapshots at different prefix lengths, which is similar to the option you mentioned, the idea here is more advanced, as these checkpoints are created automatically during the prefill and not after it. So when SillyTavern invalidates e.g. the last 1000 tokens by inserting a system message at level 4, llama.cpp can reuse one of the automatically created checkpoints and recompute only the suffix after that point.

I don't know how to better explain it, but in practice I find that, with ggml-org#20087, llama.cpp will not reprocess the prompt more often than, currently, koboldcpp. Which is why I suggested koboldcpp to also include this feature.

[LostRuins]

KoboldCpp uses its own caching algorithm, it caches during prefill too. It uses a 3 part cache, one done slightly before the end of prefill, one done after prefill is complete, and one done after generation is complete.

https://github.com/LostRuins/koboldcpp/blob/concedo_experimental/gpttype_adapter.cpp#L4464-L4482

Have you tried it?
It is incorrect to assume that our implement always matches upstream. Rather than spamming snapshots every N tokens, koboldcpp intelligently processes the prompt until X (currently set to 32) tokens before the end of the input. This is usually enough to remove boundary tokens as well as short inputs or trimmed portions of a user's previous input

[aoleg]

This. 32 tokens is obviously too little for when SillyTavern inserts anchoring message at depth 4 (default); that's 4 messages higher than the current prompt, which obviously breaks caching with koboldcpp in this scenario. How can we change this value in koboldcpp? Is there a command line option? A reasonable value for me personally would be 2048 or even 4096 tokens.
(I'm talking about things like Character Notes and Author Notes; they are widely used to ground the model into desired character voice, for example. They reinforce system messages, which have diminishing effect as the context grows.)

[LostRuins]

Hmm... The 3 part caching is triggered every turn. However, by the time 4 turns have passed its likely the checkpoint has been rotated out.

For shorter contexts you can use a higher number of checkpoints e.g. --smartcache 15, however for longer stories this becomes a little too RAM intensive. I will have to figure out a better snapshotting strategy.

[LostRuins]

I improved the snapshotting behavior, running it with about 8 slots should now be enough for this case in the next release

[wypiki]

I'm currently using OpenClaw with KoboldCpp, Qwen 3.5 122B with 128 GB of VRAM on Strix Halo, and SmartCache somehow most of the time doesn't keep KoboldCPP from reprocessing around 50000 tokens around every turn. RAM is maxed out pretty quickly with those smart cache slots, too. Those snapshots are a few hundred megabytes each and the NVMe SSD is capable of around 3 Gbit/s. So having around 100 slots ready to load into and out of the VRAM which shouldn't take more than a second would be very nice solution to make up for the slow prompt processing of the Strix Halo. Could that be done?

[LostRuins]

Ahh openclaw. Openclaw changes the context contents very frequently. Caching won't really work for it because it's constantly swapping different stuff

[wypiki]

With LMStudio the context caching worked really well with openclaw using checkpoints and rarely needed full reprocessing, but then unfortunately it crashed after 100.000-300.000 processed tokens every time which KoboldCPP doesn't.

[LostRuins]

I'm adjusting the snapshot logic, hopefully the next version should require less reprocessing