GGUF explained
What GGUF is, which runtimes read it, what it costs in VRAM, and all 81 of the 81 models in this index that ship in it.
What GGUF is
The most versatile format. CPU, GPU, Apple Silicon — runs everywhere. Supports hybrid inference splitting weights across RAM and VRAM. 81 of the 81 models in this index ship in it, across 7 quant levels — Q4_K_M, Q8_0, Q5_K_M, Q3_K_M, Q6_K, Q2_K, MXFP4.
What reads it
llama.cpp · Ollama. Hardware: Any — CPU / NVIDIA / AMD / Apple. Best suited to: Local / edge deployment. Where it is strong: Any hardware; CPU+GPU hybrid; Huge ecosystem; Beginner-friendly. Where it is not: Slower than GPU-native; Not ideal for high concurrency.
What it costs
Llama 3.1 8B Instruct at Q4_K_M is 4.85 bits per weight, which works out to 4.6 GB of weights plus 0.5 GB of KV cache at 4K context and a 0.5 GB activation buffer — 5.6 GB in total. The smallest card in this index that clears that comfortably is an RTX 5060 Ti 8G. Across the 79 models here with a published perplexity figure at this level, the median loss against FP16 is 2.9%.
MXFP4 is a GGUF level, not a rival format
2 models here ship weights that were released at 4 bits rather than converted down to them, distributed as GGUF at 4.25 bpw. That is why you will not find MXFP4 in the format filter: it is a quantization scheme inside this container, not a different container, and the file you download is a GGUF file. It also means there is no FP16 original to measure a perplexity loss against — the released checkpoint *is* the quantized one — and re-quantizing it to Q4_K_M costs quality for no memory saving.
GGUF levels in this index
| Level | bits/weight | Models | Median loss |
|---|---|---|---|
| Q4_K_M | 4.1–4.9 | 81 | 2.9%median of 79 |
| Q8_0 | 5.1–8.51 | 20 | 0.3%median of 19 |
| Q5_K_M | 5.68–5.71 | 17 | 1.3%median of 16 |
| Q3_K_M | 3.87 | 16 | 5.4%median of 16 |
| Q6_K | 6.56 | 10 | 0.8%median of 10 |
| Q2_K | 2.63 | 2 | 48.5%median of 2 |
| MXFP4 | 4.25 | 2 | 0.0%median of 2 |
Bits per weight is the model row's own figure, so a range means different models ship the same level at slightly different sizes. The median loss is computed only over the models with a published perplexity figure at that level; the sample size is printed beside it.
Models in this index that ship GGUF
81 of 81, largest first, each with its smallest GGUF level and that level's published perplexity loss where one exists.
- Mistral Large 3 675B Instruct675B MoE · Q3_K_M · 4.5%
- DeepSeek-V3671B MoE · Q3_K_M · 4.5%
- DeepSeek-R1671B MoE · Q3_K_M · 4.2%
- Llama 3.1 405B Instruct405B · Q3_K_M · 5.0%
- Llama 4 Maverick 17B (128E)400B MoE · Q3_K_M · 4.5%
- Qwen3 235B-A22B Instruct235B-A22B · Q3_K_M · 4.8%
- DBRX Instruct132B · Q3_K_M · 5.8%
- GPT-OSS 120B117B MoE · Q4_K_M · 1.6%
- Llama 4 Scout 17B (16E)109B MoE · Q3_K_M · 4.8%
- GLM-4.5-Air106B MoE · Q2_K · 12.0%
- Llama 3.2 90B Vision Instruct90B · Q3_K_M · 6.5%
- Qwen2.5 72B Instruct72B · Q4_K_M · 2.5%
- Llama 3.1 70B Instruct70B · Q4_K_M · 2.8%
- Llama 3.3 70B Instruct70B · Q4_K_M · 2.6%
- DeepSeek-R1-Distill-Llama-70B70B · Q4_K_M · 2.4%
- Mixtral 8x7B Instruct47B MoE · Q4_K_M · 2.8%
- Seed-OSS 36B Instruct36B · Q3_K_M · 5.6%
- Command R 35B35B · Q4_K_M · 3.0%
- Yi 1.5 34B Chat34B · Q4_K_M · 2.8%
- Qwen3 32B Instruct32B · Q3_K_M · 7.2%
- Qwen2.5 32B Instruct32B · Q3_K_M · 7.8%
- Qwen2.5-Coder 32B Instruct32B · Q4_K_M · 2.5%
- DeepSeek-R1-Distill-Qwen-32B32B · Q3_K_M · 7.2%
- Qwen3 30B-A3B Instruct30B-A3B · Q4_K_M · 2.4%
- Qwen3-Coder 30B-A3B Instruct30B-A3B · Q4_K_M · 2.3%
- Qwen3-VL 30B-A3B Instruct30B-A3B · Q3_K_M · 5.4%
- Qwen3.8 27B27B · Q4_K_M · —
- Gemma 3 27B IT27B · Q3_K_M · 5.5%
- Gemma 2 27B Instruct27B · Q4_K_M · 2.9%
- Mistral Small 24B Instruct24B · Q4_K_M · 2.9%
- Devstral Small 1.1 24B24B · Q4_K_M · 2.9%
- Magistral Small 1.2 24B24B · Q4_K_M · 2.9%
- Codestral 22B22B · Q4_K_M · 3.0%
- GPT-OSS 20B21B MoE · Q4_K_M · 1.4%
- InternLM2 20B Chat20B · Q4_K_M · 2.9%
- DeepSeek-Coder-V2-Lite Instruct16B · Q4_K_M · 3.1%
- DeepSeek-V2-Lite Chat16B · Q4_K_M · 3.0%
- StarCoder2 15B15B · Q4_K_M · 3.2%
- Qwen3 14B Instruct14B · Q4_K_M · 2.6%
- Qwen2.5 14B Instruct14B · Q4_K_M · 2.9%
- DeepSeek-R1-Distill-Qwen-14B14B · Q4_K_M · 2.8%
- Phi-4 14B14B · Q4_K_M · 2.7%
- Phi-3 Medium 14B Instruct14B · Q4_K_M · 3.0%
- Mistral Nemo 12B Instruct12B · Q4_K_M · 3.1%
- Gemma 3 12B IT12B · Q4_K_M · 2.9%
- Stable LM 2 12B Chat12B · Q4_K_M · 3.2%
- Jamba 1.5 Mini12B · Q4_K_M · 3.4%
- Llama 3.2 11B Vision Instruct11B · Q4_K_M · 3.5%
- Solar 10.7B Instruct11B · Q4_K_M · 3.0%
- Falcon 3 10B Instruct10B · Q4_K_M · 3.1%
- Gemma 2 9B Instruct9B · Q4_K_M · 3.3%
- GLM-4-9B-Chat9B · Q4_K_M · 3.0%
- Qwen3-VL 8B Instruct8B · Q4_K_M · 3.0%
- Ministral 3 8B Instruct8B · Q4_K_M · —
- Qwen2-VL 7B Instruct7B · Q4_K_M · 3.8%
- Granite 3.1 8B Instruct8B · Q4_K_M · 3.0%
- Qwen3 8B Instruct8B · Q4_K_M · 2.8%
- Llama 3.1 8B Instruct8B · Q2_K · 48.5%
- Nous Hermes 3 Llama 3.1 8B8B · Q4_K_M · 3.0%
- Aya 23 8B8B · Q4_K_M · 3.2%
- OpenChat 3.6 8B8B · Q4_K_M · 3.1%
- DeepSeek-R1-Distill-Llama-8B8B · Q4_K_M · 2.6%
- InternLM2 7B Chat7B · Q4_K_M · 3.1%
- Qwen2.5 7B Instruct7B · Q4_K_M · 3.0%
- Qwen2.5-Coder 7B Instruct7B · Q4_K_M · 2.8%
- WizardLM-2 7B7B · Q4_K_M · 3.1%
- DeepSeek-R1-Distill-Qwen-7B7B · Q4_K_M · 3.0%
- OLMo 2 7B Instruct7B · Q4_K_M · 3.4%
- Mistral 7B Instruct v0.37B · Q4_K_M · 3.2%
- Zephyr 7B Beta7B · Q4_K_M · 3.3%
- Gemma 3 4B IT4B · Q4_K_M · 3.2%
- Qwen3 4B Instruct4B · Q4_K_M · 2.9%
- Phi-4 Mini Instruct3.8B · Q4_K_M · 3.5%
- Phi-3.5 Mini Instruct3.8B · Q4_K_M · 3.8%
- Llama 3.2 3B Instruct3B · Q4_K_M · 3.5%
- Qwen2.5 3B Instruct3B · Q4_K_M · 3.2%
- Gemma 2 2B Instruct2B · Q4_K_M · 3.8%
- Qwen3 1.7B Instruct1.7B · Q4_K_M · 3.0%
- Qwen2.5 1.5B Instruct1.5B · Q4_K_M · 4.0%
- Llama 3.2 1B Instruct1B · Q4_K_M · 4.2%
- Qwen2.5 0.5B Instruct0.5B · Q4_K_M · 5.2%
Common questions
What is GGUF?
The most versatile format. CPU, GPU, Apple Silicon — runs everywhere. Supports hybrid inference splitting weights across RAM and VRAM. It is read by llama.cpp · Ollama, needs Any — CPU / NVIDIA / AMD / Apple, and 81 of the 81 models in this index ship in it.
How much VRAM does Llama 3.1 8B Instruct need in GGUF?
About 5.6 GB at Q4_K_M with a 4K context window — 4.6 GB of that is the weights themselves, so an 8 GB card clears it comfortably. Longer context adds KV cache on top; the calculator will size any combination.
Which GGUF level should I download?
This index carries Q4_K_M (81 models), Q8_0 (20 models), Q5_K_M (17 models), Q3_K_M (16 models), Q6_K (10 models), Q2_K (2 models), MXFP4 (2 models). Q4_K_M is the one most models here ship and the usual starting point. Q2_K at 2.63 bpw is the smallest and Q8_0 at 8.51 bpw the largest — the rule that matters is to take the highest level that still leaves headroom on your card, not the smallest one that loads.
GGUF or something else?
Depends on what reads it — this one belongs where the job is local / edge deployment. This site compares it directly against AWQ, EXL2, GPTQ, using only the models that ship both formats — the only place the comparison is measurable rather than editorial.