What you need first
A working `vllm serve` before you tune anything — the settings below only make sense once you have a baseline to move. You also need to know what your traffic actually looks like: the longest prompt you really receive and how many requests overlap. Every knob here trades those two against each other, and tuning without knowing them is guessing.
vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ --host 127.0.0.1 --port 8000The three knobs that matter
`--max-model-len` is the one to set first and the one most people leave alone: vLLM reserves KV cache for the window you declare, so a 128K default spends memory on a context your users never send. `--gpu-memory-utilization` is the share of the whole card vLLM may claim — it is a fraction of total, not of free, so anything else on the card comes out of its budget. `--max-num-seqs` caps how many requests are batched at once, which is the direct lever on how much cache each one gets.
vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ \
--max-model-len 8192 \
--gpu-memory-utilization 0.90 \
--max-num-seqs 16 \
--host 127.0.0.1 --port 8000Check it actually worked
Tuning is only real if you can see it. The server exposes Prometheus metrics, and the two that answer "did this help" are the KV cache utilisation and the preemption counter — a counter that stays at zero under your real load is the goal, not a number you guess at.
curl -s http://127.0.0.1:8000/metrics | grep -E "kv_cache|preemption|num_requests"
# A preemption counter climbing under load means the cache is too small
# for the requests in flight — not that the GPU is too slow.What the numbers should look like
Memory first: Qwen2.5 7B at AWQ INT4 is 3.6 GB of weights, 4.2 GB in total at 4K and 5.9 GB at 32K. That difference is entirely KV cache, and it is what `--max-model-len` is spending. Throughput: a single stream cannot exceed bandwidth divided by weight size, so ~278 tok/s on a 1,008 GB/s RTX 4090 — this index measured 218 tok/s for Llama 3.1 8B AWQ under vLLM on that card. Batched throughput is higher and is the reason vLLM exists, but this site has not measured it and will not quote a multiple it did not observe.
When it does not work
Preemption warnings naming `PreemptionMode.RECOMPUTE`: the cache cannot hold the in-flight requests, so vLLM throws work away and redoes it — throughput falls off a cliff rather than degrading gently. Shorten `--max-model-len`, lower `--max-num-seqs`, or raise `--gpu-memory-utilization`, in that order. Out of memory at start-up: something else holds VRAM, and the utilisation fraction is of the whole card. Crash at start-up with chunked prefill disabled: `--max-num-batched-tokens` must exceed `--max-model-len` in that configuration. And if latency is fine but throughput is poor under concurrency, check you are not running `--enforce-eager` left over from debugging — it skips CUDA-graph capture, which is exactly the steady-state decode path you want in production.
Common questions
What should I set gpu-memory-utilization to?
High, on a card doing nothing else — 0.90 is reasonable when vLLM is the only tenant. It is a fraction of the card’s total memory rather than of what is free, so a desktop session or another process comes straight out of vLLM’s budget. Raise it to cure preemption only after shortening `--max-model-len`, which is usually the real cause.
Why is my vLLM throughput collapsing under load?
Almost always KV cache pressure. The log names it: a preemption warning mentioning `PreemptionMode.RECOMPUTE` means requests are being evicted and recomputed, which costs more than it saves. It is a memory problem rather than a compute one — shorten the declared context window first, then reduce the batch width.
Is AWQ the right format for serving?
It is what this stack is built around: AWQ quantizes with the activation distribution in hand and is a first-class citizen in vLLM. 53 of the 81 models in this index ship an AWQ build. If your model has no AWQ build, that is a real constraint rather than something a conversion solves — going from one lossy format to another stacks a second round of loss on the first.
What this guide uses
- Hardware
- RTX 409024GBRTX 3090
- Picks for it
- Best local LLM for 24GB
Next steps
See everything that fits 🟢 RTX 4090Reverse lookup — 24GB at 8192 context, ranked by qualityModels covered in this guide
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
Multi-Model API Server on RTX 4090 with vLLM
Serve multiple AWQ-quantized models with vLLM's continuous batching for production-grade throughput.
Run Llama 3.1 8B on a €20/month VPS
A complete guide to running a private LLM API on a budget Linux VPS using llama.cpp server mode.
TabbyAPI: ExLlamaV2 with a Web UI
Wrap ExLlamaV2 in TabbyAPI for a polished OpenAI-compatible server with streaming and model hot-swap.