AWQ vs EXL2: which quantization format should you use?
Compared on hardware support, runtime, quality and what the 110-model index actually ships.
At a glance
| AWQ | EXL2 | |
|---|---|---|
| Runs on | NVIDIA GPU (CUDA 11.8+) or AMD via vLLM ROCm | NVIDIA GPU (Ampere+ recommended) |
| Runtime | vLLM · AutoAWQ · TGI | ExLlamaV2 |
| Best for | High-throughput API server | Local single-GPU chat on NVIDIA |
| Models in this index | 53 / 110 | 21 / 110 |
AWQ
Activation-Aware Weight Quantization: 4-bit weights quantized with the activation statistics in hand, served mainly by vLLM on NVIDIA and on the AMD cards vLLM's ROCm build supports. On every model in this index that ships both, the AWQ INT4 file is smaller than GGUF Q4_K_M (median 18% smaller) and its published quality loss is higher (median 1.1 points more). AutoAWQ, the tool most AWQ builds were made with, is deprecated in favour of vLLM's llm-compressor; existing AWQ checkpoints still load.
Strengths
- Smaller than GGUF Q4_K_M on every paired model here
- Served by vLLM with continuous batching
- Measured 218 tok/s (Llama 3.1 8B, RTX 4090, vLLM)
Trade-offs
- Higher published loss than Q4_K_M on every paired model here
- GPU only — no CPU or Apple path
- AutoAWQ deprecated (llm-compressor replaces it)
EXL2
ExLlamaV2 format. Mixed-precision per-layer quantization with a high accuracy-per-bit ratio, and on this site's own measurements the fastest single-GPU format it has run. ExLlamaV2 itself is now archived — development moved to ExLlamaV3 and its new EXL3 format — and the servers that used to load EXL2 (TabbyAPI, text-generation-webui) now load EXL3 instead, so EXL2 runs locally through ExLlamaV2's own scripts but no longer has a maintained API server.
Strengths
- Fastest single-GPU format measured here
- High accuracy per bit
- Ultra-low 2bpw option
Trade-offs
- NVIDIA only
- Runtime archived — no maintained API server
- Steeper learning curve
Models that ship both
These 16 models publish weights in both formats, so the two rows describe the same model rather than two different ones — the only place this comparison is measurable rather than editorial.
| Model | AWQ Level | Quality loss | EXL2 Level | Quality loss |
|---|---|---|---|---|
| Qwen2.5 72B Instruct | AWQ INT4 | 3.5% | EXL2 3.5bpw | 4.8% |
| Llama 3.1 70B Instruct | AWQ INT4 | 3.9% | EXL2 3.5bpw | 5.2% |
| Qwen3 32B Instruct | AWQ INT4 | 3.6% | EXL2 3.5bpw | 4.5% |
| Qwen2.5-Coder 32B Instruct | AWQ INT4 | 3.5% | EXL2 3.5bpw | 4.5% |
| Mistral Small 24B Instruct | AWQ INT4 | 3.8% | EXL2 4.65bpw | 2.2% |
| Devstral Small 1.1 24B | AWQ INT4 | 4% | EXL2 4.65bpw | 2.5% |
| Magistral Small 1.2 24B | AWQ INT4 | 4% | EXL2 4.65bpw | 2.5% |
| Qwen3 14B Instruct | AWQ INT4 | 3.5% | EXL2 4.65bpw | 1.8% |
| Qwen2.5 14B Instruct | AWQ INT4 | 3.8% | EXL2 4.65bpw | 2.1% |
| DeepSeek-R1-Distill-Qwen-14B | AWQ INT4 | 3.6% | EXL2 4.65bpw | 2% |
| Qwen3 8B Instruct | AWQ INT4 | 3.8% | EXL2 4.65bpw | 2% |
| Llama 3.1 8B Instruct | AWQ INT4 | 4.5% | EXL2 4.65bpw | 2.5% |
| DeepSeek-R1-Distill-Llama-8B | AWQ INT4 | 3.5% | EXL2 4.65bpw | 1.9% |
| Qwen2.5 7B Instruct | AWQ INT4 | 4.2% | EXL2 4.65bpw | 2.2% |
| Qwen2.5-Coder 7B Instruct | AWQ INT4 | 4% | EXL2 4.65bpw | 2% |
| Qwen3 4B Instruct | AWQ INT4 | 3.8% | EXL2 4.65bpw | 2.1% |
Quality loss is perplexity increase against the unquantized weights — lower is better, and only comparable within one model.