Which quant level should you download?

How the pick is made

Every level the model is published at is sized with the VRAM calculator at 4K context, then judged against the card: green is at most 88% of it, amber up to 105%, and a format the card's runtime cannot read is out at any size. Of the green levels, the pick is the most faithful one: the lowest published perplexity rise when the level has one, otherwise the most bits per weight — within one model more bits is closer to the original, which is the only ordering the data supports when no perplexity was published. It is the same rule the calculator's "Best fit" uses, so the two always agree.

What the level names mean

The number is roughly the bits stored per weight. GGUF Q8_0 keeps about 8.5 bits, Q6_K about 6.6, Q5_K_M about 5.7, Q4_K_M about 4.9 and Q3_K_M about 3.9; the file shrinks in step, and the perplexity rise grows as the bits fall. AWQ INT4 and GPTQ INT4 are 4-bit formats that vLLM runs, usually a little smaller than Q4_K_M. A model released in a native low-bit format, such as GPT-OSS in MXFP4, has no full-precision original to lose against, so that level is the reference.

Common questions

Q4 or Q5 — which should I pick?

If both are green on your card at the context you need, take the higher one: it is closer to the original model and costs only memory you have spare. If only Q4 is green, take Q4 rather than a tight Q5 — a level that barely loads leaves no room for context, and the runtime may push layers to system memory, which is far slower. Where a model's perplexity is published for both levels, its own page shows the difference.

Why does the pick change when I raise the context?

The weights stay the same size at any context, but the KV cache grows with every token of context, and it shares the card with them. A level that is green at 4K can turn amber at 32K, and the pick moves down a level to make room. The context-length page shows how far each pick stretches; the calculator recomputes the pick at whatever context you set.

Why is the pick AWQ on some cards, and can I use it on Windows?

AWQ INT4 stores every layer at 4 bits, so it is smaller than GGUF Q4_K_M and can be the only level that fits a card comfortably. It runs in vLLM, which needs Linux or WSL2 — Ollama and LM Studio on native Windows load GGUF only. If that is your setup, take the best GGUF row in the table instead, even if it is amber, and keep the context short.