Will it fit
on your card?
Quantize everything.
Pick your card and what you want it for. You get models that actually fit, the memory maths behind each one, and the command to run it.
New here? Start with the 8GB guideWhat fits your card
Sized at 4K context, batch 1 — weights + KV cache + 10% activations, straight from the model index.
ExampleShowing RTX 4060 Ti 16G as an example — pick your own hardware above to see yours.
- Most capable that fitsMistral Small 24B Instruct
AWQ · INT4
- 13 GB of 16 GB
- 2.8 GB free
- 3.8% quality loss
- 78 tok/s (RTX 4090)
The largest model in the index that still fits this card comfortably at 4K context.
- Room to growFalcon 3 10B Instruct
GGUF · Q4_K_M
- 7.3 GB of 16 GB
- 8.7 GB free
- 3.1% quality loss
- 118 tok/s (RTX 4090)
The largest model that uses under 60% of the card, leaving space for much longer context.
- FastestQwen2.5 0.5B Instruct
GGUF · Q8_0
- 0.6 GB of 16 GB
- 15 GB free
- 0.5% quality loss
- 540 tok/s (RTX 4090)
Highest measured throughput of everything that fits — usually a small model. Measured on an RTX 4090, so read it as a ranking, not as your number.
Popular starting points
By hardware
Every card has its own page: what fits comfortably, what is tight, what to step up to.
- RTX 5060 Ti 16G52 models fit
- RTX 4060 Ti 8G35 models fit
- RTX 4060 Ti 16G52 models fit
- RTX 507046 models fit
- RTX 309065 models fit
- RTX 409065 models fit
- RTX 509066 models fit
- Radeon RX 9070 XT52 models fit
- Mac M4 Pro 24G52 models fit
- 32 GB RAM (CPU)65 models fit
By task
Filtered views of the index.
Start here
Measured, not guessed
A sample of the benchmark table. Every row is a run on real hardware, recorded with the stack it ran on.
| Model | Hardware | Stack | tok/s | VRAM |
|---|---|---|---|---|
| Llama 3.1 8B | RTX 4090 24G | ExLlamaV2 · EXL2 4.65bpw | 235 | 5.4 GB |
| Qwen3 30B-A3B | RTX 4090 24G | llama.cpp · GGUF Q4_K_M | 95 | 19 GB |
| Llama 3.1 8B | M3 Max 48G | Ollama · GGUF Q4_K_M | 68 | 5.7 GB |
| Llama 3.1 8B | RTX 3090 24G | ExLlamaV2 · EXL2 4.65bpw | 175 | 5.4 GB |
Method: Meta Llama 3.1 8B Instruct · WikiText-2 · 4096 ctx · batch 1
Local inference runs on RTX 4090 / 3090 / M3 Max / M2 Ultra; llama.cpp b4000+, ExLlamaV2 0.2.x, vLLM 0.6.x, Ollama 0.3.x
Full benchmark table and methodology →Data updated 2026-09-21
This week’s updates
New models, recency tags, and data cadence2026-09-21
- NewQwen3-VL 8B Instruct8B · added 2026-08-08
- NewQwen3-VL 30B-A3B Instruct30B-A3B · added 2026-08-08
- NewMagistral Small 1.2 24B24B · added 2026-08-08
- NewSeed-OSS 36B Instruct36B · added 2026-08-08
- NewMinistral 3 8B Instruct8B · added 2026-09-11
- NewQwen3.8 27B27B · added 2026-09-11
Editor's Picks
Curated standout quant releases — click to view model details
- NEWQwen3.8 27BGGUFQ4_K_M · 15.3 GB · 16 of 64 layers cache KVunslothRTX 4090 / RX 7900 XT
- NEWMinistral 3 8B InstructGGUFQ4_K_M · 5.2 GB · official Mistral GGUFmistralaiRTX 4060 Ti 8G
- NEWQwen3-VL 8B InstructGGUFQ4_K_M · 5.9 GB · vision on a 12GB cardbartowskiRTX 3060 12G
- NEWMagistral Small 1.2 24BGGUFQ4_K_M · 14.3 GB · [THINK] reasoningunslothRTX 4070 Ti Super
- NEWSeed-OSS 36B InstructGGUFQ4_K_M · 21.8 GB · 512K contextunsloth2× RTX 3090
- NEWQwen3-VL 30B-A3B InstructGGUFQ4_K_M · 19 GB · 3B active, visionbartowskiM3 Max / RTX 4090
- HOTGPT-OSS 20BGGUFMXFP4 · 12.8 GB · native 4-bitopenaiRTX 4070 Ti Super
- HOTQwen3-Coder 30B-A3B InstructGGUFQ4_K_M · 19 GB · agentic coderbartowskiRTX 4090
Before you download a 20GB file
All 25 questionsHow much VRAM do I need to run a 7B model?
About 5.1 GB at Q4_K_M with a 4K context window — 4.2 GB of weights, 0.5 GB of KV cache and a 0.5 GB activation buffer, measured on Mistral 7B Instruct v0.3 (7B). Every 8 GB card in this index clears that with room left. The figure rises with context, not with how hard you use the model: the same file at 32K needs about 9.0 GB, because the KV cache grows to 4.0 GB.
Can I run a 24B model on a 16GB card?
At 4 bits and short context, yes — Mistral Small 24B Instruct at AWQ INT4 needs about 13.2 GB of 16 GB, leaving 2.8 GB. The level matters more than the parameter count: the same model at Q4_K_M is 15.9 GB, which is 99% of the card — it loads on a card with nothing else on it and has no margin for a longer window. A 16 GB card runs 52 of the 81 models here comfortably.
How much quality do you lose at 4-bit?
At Q4_K_M the median published perplexity loss across the 79 models here that report one is 2.9% — 97.1% retained — with the range running 1.4% to 5.2%. Two caveats that matter more than the median: smaller models lose more than larger ones at the same level, and perplexity is a language-modelling measure, not a measure of whether the model still does your task. Models without a published figure are shown as a dash here rather than assumed to be lossless.
GGUF or AWQ — which should I use?
GGUF unless you are running a server. GGUF runs on anything — CPU, NVIDIA, AMD, Apple — and 81 of the 81 models here ship it, against 53 for AWQ. AWQ's case is throughput under vLLM with batched requests on an NVIDIA card; for one person talking to one model, it buys you nothing GGUF does not already do. 53 models here ship both, which is the only set where the two can be compared on the same weights.
Found a number that looks wrong?
This index is maintained by hand and checked against public model cards. Figures marked “estimated” are calculated from the architecture, not measured on a card — if one does not match what you see on your own hardware, say so and it gets corrected.
Quick Tools
VRAM Calculator
Precise memory requirements for any model × quant × context combination. Red/yellow/green hardware verdict.
CLI Generator
Generate ready-to-run llama.cpp, Ollama, vLLM, ExLlamaV2 commands. One-liner or Docker Compose.
Format Wizard
Answer 3 questions — get a personalised GGUF, AWQ, or EXL2 recommendation.
Model Compare
Side-by-side VRAM, speed, and quality for two models — find the best fit for your GPU.
Data Changelog
Last updated 2026-09-21
2026-09-21Tagged two more legacy models `superseded` (QTZ-101's small, low-risk half — its other proposal, a new cross-model `qualityScore` fed by an external leaderboard, needs a source the site owner has to pick and is not part of this): Mixtral 8x7B Instruct → Qwen3 30B-A3B (40K context vs 32K, 19GB vs 28.5GB at Q4, faster, lower measured quality loss) and Stable LM 2 12B Chat → Falcon 3 10B Instruct (32K context vs a since-dated 4K, smaller file, faster). Both reasons are the real, checked field `supersededDiffNote()` derives, not a template phrase, and both pages stay `index, follow` with their numbers intact — only `homePicks`/`/best/` stop recommending them first. Two other candidates from the same audit pass were deliberately left untagged after checking, not skipped for convenience: DeepSeek-V2-Lite Chat already carries a 163,840-token context and an 11GB Q4 footprint that nothing else in this class beats, and its only same-family relative in the index is a code-specialised sibling, not a generational upgrade — tagging it "superseded" by a model with less context is exactly the mistake the Command R 35B pick made in an earlier batch. Falcon 3 10B was the only real candidate to replace it with, which is also why it was picked as Stable LM's successor instead.
All data is manually curated and verified against community sources. Always cross-check with official Hugging Face model cards before deployment.