GGUF vs AWQ: which quantization format should you use?
Compared on hardware support, runtime, quality and what the 110-model index actually ships.
At a glance
| GGUF | AWQ | |
|---|---|---|
| Runs on | Any — CPU / NVIDIA / AMD / Apple | NVIDIA GPU (CUDA 11.8+) or AMD via vLLM ROCm |
| Runtime | llama.cpp · Ollama | vLLM · AutoAWQ · TGI |
| Best for | Local / edge deployment | High-throughput API server |
| Models in this index | 110 / 110 | 53 / 110 |
GGUF
The most versatile format. CPU, GPU, Apple Silicon — runs everywhere. Supports hybrid inference splitting weights across RAM and VRAM.
Strengths
- Any hardware
- CPU+GPU hybrid
- Huge ecosystem
- Beginner-friendly
Trade-offs
- Slower than GPU-native
- Not ideal for high concurrency
AWQ
Activation-Aware Weight Quantization: 4-bit weights quantized with the activation statistics in hand, served mainly by vLLM on NVIDIA and on the AMD cards vLLM's ROCm build supports. On every model in this index that ships both, the AWQ INT4 file is smaller than GGUF Q4_K_M (median 18% smaller) and its published quality loss is higher (median 1.1 points more). AutoAWQ, the tool most AWQ builds were made with, is deprecated in favour of vLLM's llm-compressor; existing AWQ checkpoints still load.
Strengths
- Smaller than GGUF Q4_K_M on every paired model here
- Served by vLLM with continuous batching
- Measured 218 tok/s (Llama 3.1 8B, RTX 4090, vLLM)
Trade-offs
- Higher published loss than Q4_K_M on every paired model here
- GPU only — no CPU or Apple path
- AutoAWQ deprecated (llm-compressor replaces it)
Models that ship both
These 53 models publish weights in both formats, so the two rows describe the same model rather than two different ones — the only place this comparison is measurable rather than editorial.
| Model | GGUF Level | Quality loss | AWQ Level | Quality loss |
|---|---|---|---|---|
| Llama 3.1 405B Instruct | Q4_K_M | 2.3% | AWQ INT4 | 3.5% |
| Qwen3 235B-A22B Instruct | Q4_K_M | 2.2% | AWQ INT4 | 3.2% |
| Llama 4 Scout 17B (16E) | Q4_K_M | 2.4% | AWQ INT4 | 3.2% |
| Qwen2.5 72B Instruct | Q5_K_M | 1.1% | AWQ INT4 | 3.5% |
| Llama 3.1 70B Instruct | Q5_K_M | 1.2% | AWQ INT4 | 3.9% |
| Llama 3.3 70B Instruct | Q5_K_M | 1% | AWQ INT4 | 3.7% |
| DeepSeek-R1-Distill-Llama-70B | Q4_K_M | 2.4% | AWQ INT4 | 3.5% |
| Jamba 1.5 Mini | Q4_K_M | 3.4% | AWQ INT4 | 4.8% |
| Mixtral 8x7B Instruct | Q4_K_M | 2.8% | AWQ INT4 | 3.8% |
| Seed-OSS 36B Instruct | Q5_K_M | 1.3% | AWQ INT4 | 3.8% |
| Yi 1.5 34B Chat | Q4_K_M | 2.8% | AWQ INT4 | 4% |
| Qwen3 32B Instruct | Q4_K_M | 2.5% | AWQ INT4 | 3.6% |
| Qwen2.5-Coder 32B Instruct | Q4_K_M | 2.5% | AWQ INT4 | 3.5% |
| Qwen3 30B-A3B Instruct | Q5_K_M | 1.1% | AWQ INT4 | 3.4% |
| Qwen3-Coder 30B-A3B Instruct | Q5_K_M | 1% | AWQ INT4 | 3.2% |
| Qwen3-VL 30B-A3B Instruct | Q8_0 | 0.3% | AWQ INT4 | 3.7% |
| Gemma 3 27B IT | Q4_K_M | 2.8% | AWQ INT4 | 3.8% |
| Gemma 2 27B Instruct | Q5_K_M | 1.3% | AWQ INT4 | 4% |
| Mistral Small 24B Instruct | Q4_K_M | 2.9% | AWQ INT4 | 3.8% |
| Devstral Small 1.1 24B | Q6_K | 0.6% | AWQ INT4 | 4% |
Quality loss is perplexity increase against the unquantized weights — lower is better, and only comparable within one model.