RTX 5060 — what LLMs can it run?
35 of 81 indexed models fit comfortably in 8GB at 4K context, each at the highest-quality quant that still leaves headroom.
14B · 5
| Model | Quant | Est. VRAM | Headroom | tok/s |
|---|---|---|---|---|
| Stable LM 2 12B Chat12B | AWQ INT4 | 7.01 GB | +1 GB | 142 |
| Jamba 1.5 Mini12B | AWQ INT4 | 6.82 GB | +1.2 GB | 125 |
| Solar 10.7B Instruct11B | AWQ INT4 | 6.42 GB | +1.6 GB | 168 |
| Falcon 3 10B Instruct10B | GPTQ INT4 | 6.12 GB | +1.9 GB | 155 |
| Gemma 2 9B Instruct9B | AWQ INT4 | 6.27 GB | +1.7 GB | 188 |
7B · 20
| Model | Quant | Est. VRAM | Headroom | tok/s |
|---|---|---|---|---|
| GLM-4-9B-Chat9B | Q4_K_M | 5.87 GB | +2.1 GB | 135 |
| Qwen3-VL 8B Instruct8B | Q4_K_M | 6.19 GB | +1.8 GB | 140 |
| Ministral 3 8B Instruct8B | Q5_K_M | 6.92 GB | +1.1 GB | — |
| Qwen2-VL 7B Instruct7B | Q4_K_M | 5.49 GB | +2.5 GB | 72 |
| Granite 3.1 8B Instruct8B | Q4_K_M | 5.88 GB | +2.1 GB | 142 |
| Qwen3 8B Instruct8B | EXL2 4.65bpw | 5.59 GB | +2.4 GB | 228 |
| Llama 3.1 8B Instruct8B | EXL2 4.65bpw | 5.43 GB | +2.6 GB | 235 |
| Nous Hermes 3 Llama 3.1 8B8B | EXL2 4.65bpw | 5.43 GB | +2.6 GB | 232 |
| Aya 23 8B8B | Q4_K_M | 5.64 GB | +2.4 GB | 145 |
| OpenChat 3.6 8B8B | EXL2 4.65bpw | 5.43 GB | +2.6 GB | 228 |
| DeepSeek-R1-Distill-Llama-8B8B | Q5_K_M | 6.51 GB | +1.5 GB | 128 |
| InternLM2 7B Chat7B | Q4_K_M | 5.45 GB | +2.6 GB | 148 |
| Qwen2.5 7B Instruct7B | Q6_K | 6.77 GB | +1.2 GB | 132 |
| Qwen2.5-Coder 7B Instruct7B | EXL2 4.65bpw | 4.87 GB | +3.1 GB | 248 |
| WizardLM-2 7B7B | Q4_K_M | 5.07 GB | +2.9 GB | 152 |
| DeepSeek-R1-Distill-Qwen-7B7B | EXL2 4.65bpw | 4.87 GB | +3.1 GB | 210 |
| OLMo 2 7B Instruct7B | Q4_K_M | 6.82 GB | +1.2 GB | 150 |
| Mistral 7B Instruct v0.37B | Q6_K | 6.75 GB | +1.3 GB | 135 |
| Zephyr 7B Beta7B | Q6_K | 6.75 GB | +1.3 GB | 132 |
| Gemma 3 4B IT4B | Q8_0 | 5.36 GB | +2.6 GB | 145 |
≤3B · 10
| Model | Quant | Est. VRAM | Headroom | tok/s |
|---|---|---|---|---|
| Qwen3 4B Instruct4B | Q6_K | 4.05 GB | +4 GB | 145 |
| Phi-4 Mini Instruct3.8B | Q8_0 | 4.68 GB | +3.3 GB | 262 |
| Phi-3.5 Mini Instruct3.8B | Q8_0 | 5.89 GB | +2.1 GB | 255 |
| Llama 3.2 3B Instruct3B | Q8_0 | 4.05 GB | +4 GB | 285 |
| Qwen2.5 3B Instruct3B | Q8_0 | 3.67 GB | +4.3 GB | 290 |
| Gemma 2 2B Instruct2B | Q8_0 | 3.34 GB | +4.7 GB | 320 |
| Qwen3 1.7B Instruct1.7B | Q8_0 | 2.39 GB | +5.6 GB | 240 |
| Qwen2.5 1.5B Instruct1.5B | Q8_0 | 1.83 GB | +6.2 GB | 410 |
| Llama 3.2 1B Instruct1B | Q8_0 | 1.51 GB | +6.5 GB | 450 |
| Qwen2.5 0.5B Instruct0.5B | Q8_0 | 0.6 GB | +7.4 GB | 540 |
How this list is built
Each row is the lowest-perplexity-loss quant of that model whose estimated total — weights plus KV cache at 4K context plus activation buffer — uses at most 88% of the card. That is the calculator's "green" threshold, so every row here has real headroom rather than only just fitting. Raise the context length and the list shortens; the calculator lets you check any combination directly.
A further 5 models load but with no headroom to spare (up to 105% of VRAM) — the Quant Hub’s GPU chips count those too, which is why its number is higher.
Measured on this card
No benchmark runs in this index were recorded on a RTX 5060. Every figure on this page is calculated from the model architecture and the quant level — treat them as estimates, not measurements.
Cards with the same budget
What fits is decided by memory, so every 8GB card of this type returns the same list. These pages are not different answers — they differ in throughput, which this index does not measure per card.
Stepping up
A RTX 3080 10G (10GB) fits 9 more of the indexed models than this card. RTX 3080 10G →
35 of 81 indexed models fit comfortably in 8GB at 4K context, each at the highest-quality quant that still leaves headroom.