CUDA error: out of memory (llama.cpp)

The message

CUDA error: out of memory

What it means

llama.cpp asked the card for memory and none was left: the weights, the KV cache for your context and the compute buffers add up to more than the free VRAM.

Why it happens

• The model at this quant level, at this context, does not fit the card. Context is the part people forget: the KV cache grows with every token of -c.
• You passed -ngl all (or a large -ngl) or a large -c. llama.cpp's automatic fitting (--fit, on by default) only adjusts arguments you did not set, so a forced value can push it over.
• Something else holds VRAM: another model server, a game, the desktop on the same card. nvidia-smi lists every process and what it uses.
• On AMD builds the same failure prints ROCm error: out of memory — the cause and fixes are the same.

How to fix it

1. Check the size first: put the model, level and context into the VRAM calculator with your card. If it is red, no flag will make it fit — pick a smaller level or a shorter context.
2. Shorten the context: -c 8192 instead of the model's full window. Omitting -c starts from the context the model was trained for.
3. Quantize the KV cache: -ctk q8_0 -ctv q8_0 roughly halves it, with a very small quality cost.
4. Offload fewer layers: drop -ngl (or set a number below the layer count) so the rest runs from system RAM. It loads, but generation slows sharply. For MoE models --n-cpu-moe N moves only expert weights and costs less speed.

Guides that cover this

• 8GB GPU Starter Guide: 3060 / 4060 / 3070
• llama.cpp on Windows with CUDA
• Running 70B on Dual RTX 3090 with llama.cpp

Related errors

• unable to allocate CUDA0 buffer (llama.cpp)
• failed to allocate buffer for kv cache