failed to allocate buffer for kv cache

The message

failed to allocate buffer for kv cache

What it means

The weights loaded, but there was no room left for the KV cache at the context you asked for. It is usually followed by "failed to initialize the context".

Why it happens

• The context (-c, or Ollama's num_ctx) is too long for what the weights left free.

How to fix it

1. Lower -c. The context-length page lists how far common models go on common cards, and the calculator finds the longest context for any pair.
2. Quantize the cache with -ctk q8_0 -ctv q8_0 (Ollama: OLLAMA_KV_CACHE_TYPE=q8_0) to fit about twice the context.
3. Or use a smaller quant level, which leaves more room for the cache.

Related errors

• CUDA error: out of memory (llama.cpp)