How much context fits on your card?

Why context costs memory

Every token of context leaves a key and a value vector in every layer that keeps a cache: 2 × layers × kv_heads × head_dim × bytes per token. Two models of the same size can differ several-fold here. Grouped-query attention shares one KV head between several attention heads; sliding-window layers keep only a fixed window; hybrid models such as Qwen3.5/3.6 keep a growing cache in a quarter of their layers. That is why some models reach their full window on a card where a same-size model stops at a few thousand tokens. Each figure here is found by running the calculator itself at every step, not by inverting the formula.

Quantizing the KV cache

The cache does not have to be fp16. ggml's q8_0 stores 32 values in 34 bytes, about 53% of fp16, and q4_0 stores them in 18, about 28%. Ollama's documentation describes q8_0 as a very small loss in precision that usually has no noticeable impact, and q4_0 as a small-to-medium loss that may be more noticeable at long contexts, varying by model. Both runtimes need Flash Attention for it. In llama.cpp pass -ctk q8_0 -ctv q8_0 (the default -fa auto turns Flash Attention on); in Ollama set OLLAMA_KV_CACHE_TYPE=q8_0 on the server, which applies to every model it loads.

Common questions

Is a q8_0 cache worth it?

For long documents and code, usually yes: it roughly doubles the context that fits for the cache's share of memory, and Ollama's own documentation calls the precision loss very small. q4_0 stretches further but its loss is larger and more visible at long contexts, so try it on your own task before relying on it.

Why does a model stop at "full window" even with memory to spare?

Because that is the longest context the model was trained or configured for; this page never sizes a model past its own window. Runtimes may let you set a longer context, but the model was not built to use it, and answers past the window degrade.

Does llama.cpp use this much context by default?

Recent llama.cpp starts from the model's trained context when you omit -c, then trims it to fit the card, and Ollama picks its own default. Set the context yourself (-c 32768 in llama.cpp, num_ctx in Ollama) so you get the length you planned for rather than whatever the runtime settled on.