Quantization format comparisons
A quantization format bundles two decisions: how the weights are stored, and which runtime can read them. Getting it wrong usually means a file that downloads fine and then refuses to load. Below is what each format is, what reads it, and — because this index records the format of every model it tracks — how likely you are to find your model in it.
The formats, counted against this index
| Format | Reads it | Runs on | Models here | Median published loss | Levels carried |
|---|---|---|---|---|---|
| GGUF | llama.cpp · Ollama | Any — CPU / NVIDIA / AMD / Apple | 110 / 110 | 3.0%median of 100 | Q4_K_M, Q8_0, Q5_K_M, Q3_K_M |
| AWQ | vLLM · AutoAWQ · TGI | NVIDIA GPU (CUDA 11.8+) or AMD via vLLM ROCm | 53 / 110 | 4.0%median of 53 | INT4 |
| EXL2 | ExLlamaV2 | NVIDIA GPU (Ampere+ recommended) | 21 / 110 | 2.3%median of 21 | 4.65bpw, 3.5bpw |
| GPTQ | GPTQModel · vLLM · TGI | NVIDIA GPU (CUDA) or AMD via vLLM ROCm | 4 / 110 | 4.8%median of 4 | INT4 |
| HQQ | HQQ · bitsandbytes | NVIDIA GPU, AMD ROCm | 0 / 110 | none published | — |
No model in this index ships this format. It is documented here as reference, not as something you will find in the Hub.
Which one should you use?
Start from what can read the file, not from what scores best. On a Mac, on CPU, on AMD, or if you simply want it to work, the answer is GGUF — it is the only format in this index that runs on every hardware class, and 110 of the 110 models here ship it. If you are serving many requests at once on NVIDIA, AWQ under vLLM is the batch-throughput answer and 53 models ship it. If you want the fastest single stream on one NVIDIA card and do not mind a narrower model selection, EXL2 covers 21. GPTQ is the older NVIDIA path at 4 models — AWQ or EXL2 does the same job better. The quality column is a median of published perplexity losses at 4-bit-class levels, not a benchmark this site ran; differences of a point or two are inside the noise of how each model was quantized.
Head to head
Format Heat Index
Editorial ranking of what people reach for — an opinion, not a measured share
- 1GGUF
- 2AWQ
- 3EXL2
- 4GPTQ
- 5HQQ
Editorial judgement from which formats Hugging Face uploads and community threads favour. It is not measured, so it is shown as an order, not a percentage — the per-format model counts above are the measured part.