What you need first
An NVIDIA card with enough memory for the model plus its KV cache, a recent driver, and Python 3.12. vLLM is a server, not a chat app — reach for it when you want an OpenAI-compatible endpoint serving concurrent requests, and reach for llama.cpp or Ollama when you want one person talking to one model.
# uv is the documented install path now; it picks the right torch build
# from your installed CUDA driver.
pip install --upgrade uv
uv pip install vllm --torch-backend=auto
vllm --versionServe the model
The entrypoint is the `vllm serve` CLI. The older `python -m vllm.entrypoints.openai.api_server` invocation is what most tutorials still show and is no longer the documented form. vLLM reads the quantization method out of the model config, so `--quantization` is not something you normally pass for an AWQ checkpoint.
vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ \
--max-model-len 32768 \
--gpu-memory-utilization 0.85 \
--host 127.0.0.1 --port 8000
# Bind to 127.0.0.1 unless you mean to expose the machine. The default
# server has no authentication of its own.Check it actually worked
The server speaks the OpenAI protocol, so the first check is the models endpoint — it answers only once weights are loaded and the KV cache is allocated, which is also the slow part of startup. Then send one completion.
curl http://127.0.0.1:8000/v1/models
curl http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"Qwen/Qwen2.5-7B-Instruct-AWQ","messages":[{"role":"user","content":"hi"}]}'What the numbers should look like
Qwen2.5 7B at AWQ INT4 is 3.6 GB of weights and needs 4.2 GB in total at 4K context, rising to 5.9 GB at 32K — the weights do not move, the KV cache does. On an RTX 4090 at 1,008 GB/s that puts a single-stream ceiling near 278 tok/s. This index has one measured vLLM run to compare against: Llama 3.1 8B at AWQ INT4 on a 4090, at 218 tok/s. Batched throughput is a different number entirely and this site has not measured it — continuous batching is the reason to run vLLM, but any specific multiple you read for it came from someone else’s hardware.
When it does not work
The server logs a preemption warning mentioning `PreemptionMode.RECOMPUTE` and throughput collapses: there is not enough KV cache space for the requests in flight. Raise `--gpu-memory-utilization`, or lower `--max-num-seqs` so fewer requests are batched, or shorten `--max-model-len` — that last one is what most people actually need, because a 128K window reserves cache for a context nobody sends. Out of memory at start-up rather than under load: `--gpu-memory-utilization` is a fraction of the whole card, so anything else holding VRAM comes out of vLLM’s share. Startup takes minutes: that is compilation and CUDA-graph capture; `--enforce-eager` skips both and costs steady-state decode speed, which is a good trade while you are iterating and a bad one in production.
Common questions
What is the current command to start a vLLM server?
`vllm serve <model-repo>`. The `python -m vllm.entrypoints.openai.api_server` form that most tutorials still show is no longer the documented entrypoint. Install with `uv pip install vllm --torch-backend=auto`, which selects the torch build matching your installed CUDA driver.
Does vLLM only run on NVIDIA?
No — that is a common claim and it is wrong. vLLM publishes official ROCm wheels (`uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/`) and ROCm Docker images, alongside backends for Intel XPU and TPU. CUDA is the best-trodden path, not the only one.
Why does vLLM reserve so much VRAM before I send a request?
By design. It pre-allocates the KV cache up front — `--gpu-memory-utilization` is the fraction of the whole card it may take — so that batching never has to allocate mid-request. That is why a 128K `--max-model-len` costs memory even when every request is 2K long, and why shortening it is usually the first fix for preemption warnings.
What this guide uses
- Hardware
- RTX 409024GBRTX 3090
- Models
- Qwen2.5 7B Instruct7B
- Picks for it
- Best local LLM for 24GB
Next steps
See everything that fits 🟢 RTX 4090Reverse lookup — 24GB at 8192 context, ranked by qualityModels covered in this guide
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
vLLM + AWQ in Production: Tuning Guide
gpu-memory-utilization, max-model-len, and batching knobs for stable API serving.
TabbyAPI: ExLlamaV2 with a Web UI
Wrap ExLlamaV2 in TabbyAPI for a polished OpenAI-compatible server with streaming and model hot-swap.
Nginx Reverse Proxy for Local LLM APIs
Put Ollama or llama.cpp behind Nginx with TLS, rate limiting, and a stable /v1 endpoint for your apps.