GGUF vs EXL2: which quantization format should you use?
Compared on hardware support, runtime, quality and what the 110-model index actually ships.
At a glance
| GGUF | EXL2 | |
|---|---|---|
| Runs on | Any — CPU / NVIDIA / AMD / Apple | NVIDIA GPU (Ampere+ recommended) |
| Runtime | llama.cpp · Ollama | ExLlamaV2 |
| Best for | Local / edge deployment | Local single-GPU chat on NVIDIA |
| Models in this index | 110 / 110 | 21 / 110 |
GGUF
The most versatile format. CPU, GPU, Apple Silicon — runs everywhere. Supports hybrid inference splitting weights across RAM and VRAM.
Strengths
- Any hardware
- CPU+GPU hybrid
- Huge ecosystem
- Beginner-friendly
Trade-offs
- Slower than GPU-native
- Not ideal for high concurrency
EXL2
ExLlamaV2 format. Mixed-precision per-layer quantization with a high accuracy-per-bit ratio, and on this site's own measurements the fastest single-GPU format it has run. ExLlamaV2 itself is now archived — development moved to ExLlamaV3 and its new EXL3 format — and the servers that used to load EXL2 (TabbyAPI, text-generation-webui) now load EXL3 instead, so EXL2 runs locally through ExLlamaV2's own scripts but no longer has a maintained API server.
Strengths
- Fastest single-GPU format measured here
- High accuracy per bit
- Ultra-low 2bpw option
Trade-offs
- NVIDIA only
- Runtime archived — no maintained API server
- Steeper learning curve
Models that ship both
These 21 models publish weights in both formats, so the two rows describe the same model rather than two different ones — the only place this comparison is measurable rather than editorial.
| Model | GGUF Level | Quality loss | EXL2 Level | Quality loss |
|---|---|---|---|---|
| Qwen2.5 72B Instruct | Q5_K_M | 1.1% | EXL2 3.5bpw | 4.8% |
| Llama 3.1 70B Instruct | Q5_K_M | 1.2% | EXL2 3.5bpw | 5.2% |
| Qwen3 32B Instruct | Q4_K_M | 2.5% | EXL2 3.5bpw | 4.5% |
| Qwen2.5 32B Instruct | Q4_K_M | 2.7% | EXL2 3.5bpw | 4.8% |
| Qwen2.5-Coder 32B Instruct | Q4_K_M | 2.5% | EXL2 3.5bpw | 4.5% |
| DeepSeek-R1-Distill-Qwen-32B | Q4_K_M | 2.6% | EXL2 3.5bpw | 4.5% |
| Mistral Small 24B Instruct | Q4_K_M | 2.9% | EXL2 4.65bpw | 2.2% |
| Devstral Small 1.1 24B | Q6_K | 0.6% | EXL2 4.65bpw | 2.5% |
| Magistral Small 1.2 24B | Q6_K | 0.6% | EXL2 4.65bpw | 2.5% |
| Qwen3 14B Instruct | Q5_K_M | 1.2% | EXL2 4.65bpw | 1.8% |
| Qwen2.5 14B Instruct | Q5_K_M | 1.4% | EXL2 4.65bpw | 2.1% |
| DeepSeek-R1-Distill-Qwen-14B | Q4_K_M | 2.8% | EXL2 4.65bpw | 2% |
| Qwen3 8B Instruct | Q6_K | 0.6% | EXL2 4.65bpw | 2% |
| Llama 3.1 8B Instruct | Q8_0 | 0.1% | EXL2 4.65bpw | 2.5% |
| Nous Hermes 3 Llama 3.1 8B | Q4_K_M | 3% | EXL2 4.65bpw | 2.3% |
| OpenChat 3.6 8B | Q4_K_M | 3.1% | EXL2 4.65bpw | 2.4% |
| DeepSeek-R1-Distill-Llama-8B | Q5_K_M | 1.2% | EXL2 4.65bpw | 1.9% |
| Qwen2.5 7B Instruct | Q6_K | 0.7% | EXL2 4.65bpw | 2.2% |
| Qwen2.5-Coder 7B Instruct | Q4_K_M | 2.8% | EXL2 4.65bpw | 2% |
| DeepSeek-R1-Distill-Qwen-7B | Q4_K_M | 3% | EXL2 4.65bpw | 2.2% |
Quality loss is perplexity increase against the unquantized weights — lower is better, and only comparable within one model.