CUDA out of memory. Tried to allocate (vLLM)

The message

CUDA out of memory. Tried to allocate <n> GiB. GPU 0 has a total capacity of <n> GiB of which <n> MiB is free.

Parts in <angle brackets> change from run to run.

What it means

PyTorch, which vLLM runs on, could not allocate the next tensor. In vLLM it usually means the model plus its KV cache does not fit at your settings.

Why it happens

• The weights at this format do not fit, or leave too little for vLLM's KV cache and CUDA graphs.
• --max-model-len is long, so the minimum cache vLLM needs is large.
• Another process holds part of the card.

How to fix it

1. Shorten --max-model-len first; raise --gpu-memory-utilization only after that, and not past what the card can actually give.
2. Use a 4-bit AWQ or GPTQ build of the model; check it in the calculator against your card.
3. --enforce-eager skips CUDA graph capture, which frees some memory at a cost in speed.

Guides that cover this

• vLLM + AWQ in Production: Tuning Guide
• Multi-Model API Server on RTX 4090 with vLLM

Related errors

• vLLM: free memory less than GPU memory utilization