What fits comfortably
At 4K context, 8GB cards handle 7–8B models at Q4_K_M with headroom. Push to 3B at Q8 or 14B at Q3 only if you lower context to 2K. Avoid 32B+ entirely on 8GB.
text
✓ Qwen2.5 7B Q4_K_M → ~7.4 GB @ 4K ctx
✓ Llama 3.1 8B Q4_K_M → ~7.7 GB @ 4K ctx
✓ Phi-4 Mini Q4_K_M → ~4.8 GB @ 4K ctx
△ DeepSeek-R1 14B Q3_K_M → ~9.5 GB @ 2K ctx (tight)
✗ Qwen2.5 32B any quant → needs 16GB+Quick start with Ollama
Ollama auto-picks a quant for your VRAM. Start small, verify speed, then try a larger model. Use /api/ps to check live VRAM usage.
bash
ollama pull qwen2.5:7b
ollama run qwen2.5:7b
# Check loaded models and VRAM
curl http://localhost:11434/api/psRelated guides
IntermediateEdge / Local9 min read
Run GPT-OSS 20B (and 120B) locally without re-quantizing
GPT-OSS ships natively in MXFP4, so the usual "download the Q4_K_M" habit makes it bigger and worse. Sizing, the right flags, and how MoE expert-offload puts the 120B on a 24GB card.
BeginnerEdge / Local7 min read
What Can You Run on RTX 4060 Ti 16G?
A practical guide to picking the right model and quant level for NVIDIA's best budget 16GB card.
BeginnerEdge / Local9 min read
llama.cpp on Windows with CUDA
Build llama.cpp with NVIDIA GPU support on Windows 11 — the path of least resistance for PC gamers.
Deployment guides are educational. Each model is subject to its own license — read the official Hugging Face model card before downloading or deploying.