unable to allocate CUDA0 buffer (llama.cpp)

The message

unable to allocate CUDA0 buffer

What it means

llama.cpp could not reserve room for the model's weights on that device. The device name follows your backend: CUDA0, ROCm0, Vulkan0 or Metal.

Why it happens

• The weights alone are larger than the free memory on that device — before any context is added.
• Too many layers were forced onto the card with -ngl.
• Another process already holds part of the card.

How to fix it

1. Compare the file size with your card in the VRAM calculator; the weights are roughly params × bits ÷ 8.
2. Download a smaller level of the same model, or let llama.cpp choose the layer split by not passing -ngl.
3. Free the card: stop other servers (nvidia-smi on NVIDIA, rocm-smi on AMD shows what is using it).

Guides that cover this

• What to Run on a 12GB GPU (RTX 3060 12G, 4070, 5070)
• AMD GPU + llama.cpp via ROCm (Quick Start)

Related errors

• CUDA error: out of memory (llama.cpp)
• Metal: failed to allocate buffer on a Mac