I have an NVIDIA card. Should I always use EXL2?

Only if speed is the priority and you are serving one request at a time. GGUF has far wider tooling and is the only option if the model does not fully fit in VRAM; AWQ through vLLM wins on batched throughput. The wizard weights these by the priority you pick.

What does bits-per-weight actually control?

Size and quality, jointly. Fewer bits means a smaller file and more perplexity loss, and the relationship is not linear — the drop from 4-bit to 3-bit costs far more quality than 8-bit to 6-bit. The perplexity chart on the benchmarks page shows where the curve turns.

Can I convert between formats myself?

You quantize from the original FP16 weights rather than converting between quantized formats — going from one lossy format to another compounds the loss. The cookbook has a guide for producing your own GGUF.

Why is HQQ not offered?

The format reference documents it, but no model in this index ships HQQ weights. Recommending a format with nothing behind it would send you to a page with no results.

🧭 Format Wizard

Which Quant Format Should I Use?

Answer three questions and get a data-driven recommendation for your hardware and use case.

1.What hardware are you running on?

2.What matters most to you?

3.What will you use it for?

Answer the questions and click Get Recommendation

The formats, in one paragraph each

GGUF runs everywhere — CPU, NVIDIA, AMD, Apple — and is the only format that splits a model across VRAM and system RAM. EXL2 is the fastest on consumer NVIDIA cards and lets you pick a fractional bits-per-weight, but it is CUDA-only. AWQ is built for batched serving through vLLM. GPTQ predates AWQ and covers a similar niche with wider legacy tooling. The wizard only recommends formats an indexed model actually ships.

Why hardware decides more than preference

Most of the scoring is elimination, not taste. ExLlamaV2 is CUDA-only, so EXL2 is not a slow option on a Radeon — it is not an option. AWQ and GPTQ kernels are CUDA-first with partial, version-sensitive ROCm support. On Apple silicon, GGUF through llama.cpp's Metal backend is the only path with full support. That is why an AMD or Mac answer converges on GGUF regardless of what you say you care about.

Quant levels belong to one format

Q4_K_M is a GGUF level. 4.65bpw is an EXL2 level. INT4 is what AWQ and GPTQ ship. They are not interchangeable names for the same thing, and a recommendation that mixes them — "EXL2 · Q4_K_M" — is telling you to fetch something that does not exist. Each row here names a level that belongs to its own format.

Common questions

I have an NVIDIA card. Should I always use EXL2?: Only if speed is the priority and you are serving one request at a time. GGUF has far wider tooling and is the only option if the model does not fully fit in VRAM; AWQ through vLLM wins on batched throughput. The wizard weights these by the priority you pick.
What does bits-per-weight actually control?: Size and quality, jointly. Fewer bits means a smaller file and more perplexity loss, and the relationship is not linear — the drop from 4-bit to 3-bit costs far more quality than 8-bit to 6-bit. The perplexity chart on the benchmarks page shows where the curve turns.
Can I convert between formats myself?: You quantize from the original FP16 weights rather than converting between quantized formats — going from one lossy format to another compounds the loss. The cookbook has a guide for producing your own GGUF.
Why is HQQ not offered?: The format reference documents it, but no model in this index ships HQQ weights. Recommending a format with nothing behind it would send you to a page with no results.