What you need first
An RTX 4090 or another 24GB card, and a decision about which quant format before you download anything — the two shipped builds land in different verdict bands on this card, which the rest of this guide is about.
nvidia-smi --query-gpu=name,memory.total --format=csvThe quant choice actually matters here
GGUF Q4_K_M is 18.7 GB of weights and 21.7 GB in total at 4K context — 90% of a 24GB card, which this index calls tight rather than comfortable. It loads and runs, with no margin for a longer window. AWQ INT4 is smaller at the same nominal bit depth (weights compress differently per format) at 18.1 GB total, 75% of the card — comfortable, with room to grow. If your workflow needs more than 4K of context, that difference is the one that decides whether the model fits at all.
# GGUF: simplest path, tight at 4K on a 24GB card
ollama pull qwen2.5-coder:32b
# AWQ: more headroom, needs vLLM
vllm serve Qwen/Qwen2.5-Coder-32B-Instruct-AWQ --max-model-len 8192Check it actually ran on the GPU
At 90% of the card, GGUF Q4_K_M leaves little slack for a display or a second process — if anything else is using VRAM, this is the model that will not load rather than one that degrades gracefully. Watch memory at load time, not just after.
nvidia-smi --query-gpu=memory.used,utilization.gpu --format=csv -l 1
# llama.cpp: confirm every layer offloaded
# load_tensors: offloaded 65/65 layers to GPUWhat the numbers should look like
This index has a measured run on the base Qwen2.5 32B (same parameter count and architecture family, so the memory-bandwidth story is the same): 44 tok/s at GGUF Q4_K_M on an RTX 4090. That is not a Coder-specific measurement, but the roofline it implies — bandwidth divided by weight size — applies to the Coder build too, since the file sizes are nearly identical. AWQ INT4’s smaller weight size gives it a slightly higher ceiling, though this index has no measured AWQ run at this size to compare against.
When it does not work
Out of memory loading GGUF Q4_K_M with anything else running: that is expected at 90% of the card — close other GPU consumers or switch to AWQ. Out of memory only once you raise the context: the GGUF build has almost no room for KV cache growth; AWQ buys headroom but even it does not fit at 32K on this card (107%). Slow prompt processing on long files: that is expected on a 32B model regardless of format — it is compute-bound on prefill, not something a quant level changes. And if you need long-context coding work on one 24GB card, the honest answer is a smaller model at a higher quant, not this one pushed past its comfortable range.
Common questions
Does Qwen2.5-Coder 32B fit comfortably on an RTX 4090?
It depends on the format. GGUF Q4_K_M is 21.7 GB at 4K context — 90% of the card, which this index calls tight rather than comfortable. AWQ INT4 is 18.1 GB — 75%, comfortable, with room for a somewhat longer window. Neither fits 32K context on this card.
AWQ or GGUF for Qwen2.5-Coder 32B?
AWQ if you want headroom and are willing to run vLLM; GGUF if you want the simpler Ollama/llama.cpp path and can live with 4K context and no margin. The published perplexity loss is similar between them (2.5% GGUF, 3.5% AWQ against FP16), so the decision is about memory and tooling, not quality.
How many tokens per second should I expect?
This index measured 44 tok/s for the same-size base Qwen2.5 32B at GGUF Q4_K_M on an RTX 4090 — a reasonable proxy for the Coder build, since the file sizes and therefore the bandwidth ceiling are nearly identical. There is no measured AWQ run at this size here, though its smaller weight size implies a somewhat higher ceiling.
What this guide uses
- Hardware
- RTX 409024GBRTX 3090
- Picks for it
- Best local LLM for 24GB
Next steps
See everything that fits 🟢 RTX 4090Reverse lookup — 24GB at 4096 context, ranked by qualityModels covered in this guide
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
DeepSeek-R1 Distill 14B: EXL2 vs GGUF
Head-to-head on RTX 4090 — when to pick turboderp EXL2 over bartowski GGUF.
ExLlamaV2 on RTX 4090: Full Setup Guide
Install ExLlamaV2, load an EXL2 quant, and serve an OpenAI-compatible API in under 10 minutes.
Run GPT-OSS 20B (and 120B) locally without re-quantizing
GPT-OSS ships natively in MXFP4, so the usual "download the Q4_K_M" habit makes it bigger and worse. Sizing, the right flags, and how MoE expert-offload puts the 120B on a 24GB card.