Local LLM GPU Requirements by Model Size
Compare five GPUs across model sizes at 4K context. Each populated cell links to a model and its quantization details.
Memory estimates at 4,096 tokens for one text sequence.
Each cell shows one catalog model with headroom, its quantization and its estimated memory. A dash means no qualifying example was found in that size class.
| GPU | ≤3B | 7B | 14B | 32B | 70B+ |
|---|---|---|---|---|---|
| RTX 4060 Ti 8G | Qwen3 4B Instruct GGUF · Q6_K · 4.0 GiB | GLM-4-9B-Chat GGUF · Q4_K_M · 5.9 GiB | Falcon 3 10B Instruct GPTQ · INT4 · 6.1 GiB | — | — |
| RTX 3060 12G | Qwen3 4B Instruct GGUF · Q6_K · 4.0 GiB | GLM-4-9B-Chat GGUF · Q8_0 · 10.2 GiB | DeepSeek-Coder-V2-Lite Instruct GGUF · Q4_K_M · 10.1 GiB | Hy-Memory-30B-A3B GGUF · Q8_0-expsIQ2_XXS · 9.9 GiB | — |
| RTX 4060 Ti 16G | Qwen3 4B Instruct GGUF · Q6_K · 4.0 GiB | GLM-4-9B-Chat GGUF · Q8_0 · 10.2 GiB | DeepSeek-Coder-V2-Lite Instruct GGUF · Q4_K_M · 10.1 GiB | Xing4.0 29B-A4B GGUF · Q2_K · 13.6 GiB | — |
| RTX 4090 | Qwen3 4B Instruct GGUF · Q6_K · 4.0 GiB | GLM-4-9B-Chat GGUF · Q8_0 · 10.2 GiB | DeepSeek-Coder-V2-Lite Instruct GGUF · Q8_0 · 17.6 GiB | Recursive Flint 40B-A2.5B GGUF · Q3_K_M · 20.6 GiB | Kolibri-1 78B-A3.5B GGUF · IQ1_M · 20.9 GiB |
| RTX 5090 | Qwen3 4B Instruct GGUF · Q6_K · 4.0 GiB | GLM-4-9B-Chat GGUF · Q8_0 · 10.2 GiB | DeepSeek-Coder-V2-Lite Instruct GGUF · Q8_0 · 17.6 GiB | Recursive Flint 40B-A2.5B GGUF · Q4_K_M · 26.7 GiB | Kolibri-1 78B-A3.5B GGUF · IQ2_XS · 25.6 GiB |