What you need first
An NVIDIA card with at least 10 GB free — both builds fit comfortably on a 24GB RTX 4090 with plenty to spare, and either format also fits a 16GB card. The choice here is not about whether it fits; it is about speed versus reach, which the rest of this guide measures rather than asserts.
nvidia-smi --query-gpu=memory.total --format=csvWhat each build actually costs
EXL2 at 4.65bpw is 8.1 GB of weights, 9.8 GB in total at 4K context. GGUF Q4_K_M is 8.5 GB of weights, 10.1 GB in total — a small difference despite the same nominal bit depth, because the two formats quantize differently under the hood. Both are comfortable on anything 16GB or larger.
# EXL2 via ExLlamaV2
huggingface-cli download turboderp/DeepSeek-R1-Distill-Qwen-14B-exl2 \
--revision 4.65bpw --local-dir ./models/r1-14b-exl2
# GGUF via llama.cpp / Ollama
ollama pull deepseek-r1:14bCheck it actually ran on the GPU
ExLlamaV2 has no CPU fallback and fails loudly if it does not fit, which is its own kind of confirmation. For the GGUF/Ollama path, check the placement explicitly — Ollama silently uses the CPU for whatever does not fit, and a reasoning model that is quietly running on the CPU looks like it is "thinking" rather than like it is broken.
ollama ps
# PROCESSOR should read 100% GPU
nvidia-smi --query-gpu=memory.used,utilization.gpu --format=csv -l 1What the numbers should look like
This index has a measured EXL2 run: 128 tok/s on an RTX 4090, which is close to the bandwidth-derived ceiling for 8.1 GB of weights on a 1,008 GB/s card (≈124 tok/s — measured slightly above a rounded ceiling is normal variance in the estimate, not an error). There is no measured GGUF run for this specific model at this size, so this guide will not repeat the "~95 tok/s" figure the old version quoted — nobody here ran it. The GGUF file’s slightly larger weight size (8.5 GB against 8.1 GB) implies a marginally lower ceiling, on the order of a few percent, not the ~35% gap once claimed.
When it does not work
ExLlamaV2 throws an import error rather than loading: a wheel/CUDA version mismatch — build from source with `pip install .` to match your environment exactly. GGUF via Ollama reports a CPU share: check nothing else is holding VRAM before assuming the model is too big, since both builds comfortably fit 16GB. A "reasoning" model that never seems to finish thinking: that is DeepSeek-R1’s distillation behaviour — it genuinely emits a long chain-of-thought block before the answer, on both formats, and is not a sign anything is broken.
Common questions
Is EXL2 really faster than GGUF for this model?
On this index’s own measurement, yes for EXL2 (128 tok/s on an RTX 4090) — but there is no measured GGUF run for this model here to compare against directly. The two files are close in size (8.1 GB vs 8.5 GB), so the bandwidth-derived ceilings are close too; a large gap between them would be surprising rather than expected.
Does this model fit on a 16GB card?
Yes, comfortably, in either format. GGUF Q4_K_M is 10.1 GB and EXL2 4.65bpw is 9.8 GB at 4K context — both well under a 16GB budget, with room for a longer context window than the 4K used here.
Why does DeepSeek-R1 take so long to answer?
It is a reasoning-distilled model: it generates an explicit chain-of-thought block before its final answer, which is real generated content rather than a stall. That behaviour is the same on both EXL2 and GGUF builds — it is a property of the model, not the format or the runtime.
What this guide uses
- Hardware
- RTX 409024GBRTX 3090
- Picks for it
- Best local LLM for 24GB
Next steps
See everything that fits 🟢 RTX 4090Reverse lookup — 24GB at 4096 context, ranked by qualityModels covered in this guide
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
ExLlamaV2 on RTX 4090: Full Setup Guide
Install ExLlamaV2, load an EXL2 quant, and serve an OpenAI-compatible API in under 10 minutes.
What Can You Run on RTX 4060 Ti 16G?
A practical guide to picking the right model and quant level for NVIDIA's best budget 16GB card.
Qwen2.5-Coder 32B on a Single RTX 4090
The best open coding model that fits in 24GB — quant selection and tuning tips.