vLLM: free memory less than GPU memory utilization

The message

Free memory on device <dev> (<free>/<total> GiB) on startup is less than desired GPU memory utilization (<u>, <n> GiB). Decrease GPU memory utilization or reduce GPU memory used by other processes.

Parts in <angle brackets> change from run to run.

What it means

vLLM reserves a fixed share of the card at startup (--gpu-memory-utilization, 0.92 in current releases unless you set it). Something else already uses enough of the card that this share is not free.

Why it happens

• A desktop session, a browser or another model server on the same card.
• A previous vLLM process that did not exit cleanly.

How to fix it

1. Lower the share: --gpu-memory-utilization 0.85 (this site's generated commands use 0.85).
2. Find and stop whatever holds the card with nvidia-smi.

Guides that cover this

• vLLM + AWQ in Production: Tuning Guide
• Multi-Model API Server on RTX 4090 with vLLM

Related errors

• CUDA out of memory. Tried to allocate (vLLM)