How it works

Long prompts on GGUF

On GGUF, Quail matched llama.cpp's own server everywhere except long prompts: a 4,096-token prompt reached its first token 8–24% later.

Why

The comparison reads long prompts right after the level with 8 requests at once. Quail left those earlier requests' caches in llama.cpp's shared cache. The long prompt's attention then worked across tens of thousands of positions it didn't need. llama.cpp's server moves idle requests out first.

What changed

• Earlier caches move out of the way. Quail now moves an idle request's cache out of llama.cpp's shared cache into ordinary memory, up to 8 GB as llama-server does. It moves it back if the conversation continues.
• No wasted checkpoints. It stopped saving checkpoints that a sliding-window model could never use.

Result

Keeping those caches costs memory: Quail's footprint now grows by up to about 8 GB while it serves, as llama-server's does. (#138, #144)