Who this is for
An 8GB NVIDIA card (RTX 3060 8G, 3070, 4060, 4060 Ti 8G) on Windows or Linux, with a working driver and roughly 16GB of system RAM. Every figure below is for a single stream — batch 1 — at the context length stated. The commands use Ollama; llama.cpp works the same way and is noted where it differs.
What fits, and what the numbers mean
These are estimates from this site’s VRAM calculator — model weights at the level’s measured bits-per-weight, plus the KV cache for the stated context, plus a 10% activation buffer. They are not measured file sizes and they exclude whatever your desktop is already using, which on Windows is commonly 0.5–1.5GB. That is why the green band stops at 88% of the card rather than 100%. At 4K context an 8GB card runs 7–8B at Q4_K_M with real room to spare; 14B does not fit at Q4_K_M at any context, because the weights alone are over the card.
Model Level Context Estimate On 8GB
----------------------------------------------------------------------
Phi-4 Mini 3.8B Q4_K_M 4K 2.9 GB fits, large margin
Qwen3 4B Q6_K 4K 4.1 GB fits
Qwen2.5 7B Q4_K_M 4K 5.1 GB fits
Llama 3.1 8B Q4_K_M 4K 5.6 GB fits
Qwen3 8B Q4_K_M 4K 5.8 GB fits
Llama 3.1 8B Q4_K_M 16K 7.1 GB fits, little margin
Qwen3 14B Q4_K_M 2K 9.7 GB over the card — see below
Qwen2.5 32B any 4K 21.7 GB not on 8GBWhat "over the card" actually means
A 14B at Q4_K_M needs about 9.7GB against 8GB of VRAM. That is not "tight" — it will not load fully on the GPU. Two real options: use the AWQ/GPTQ INT4 build of the same model, which the index puts at about 8.1GB and which is still above the comfortable band, or keep GGUF and offload part of the model to system RAM. Partial offload works and is what Ollama does by default when a model does not fit, but the layers left on the CPU are read over PCIe every token, so throughput falls sharply — how far depends on your PCIe link and RAM speed, and this guide has not measured it on your hardware. Budget for the offloaded weights in system RAM on top of what the OS needs: roughly 2GB for the 14B case above, and keep 16GB total system RAM as the floor.
# llama.cpp: choose the split yourself, and watch the log line
# "offloaded N/M layers to GPU" to confirm what actually landed on the card
llama-server -m qwen3-14b-Q4_K_M.gguf -ngl 28 -c 2048 --host 127.0.0.1
# Fewer layers on the GPU = less VRAM, slower generation.
# Drop -ngl until it loads, rather than guessing.Quick start with Ollama
Ollama does not pick a quantization to match your VRAM. Pulling a bare tag gives you that tag's default build — Q4_K_M for most models — on a 24GB card and an 8GB one alike; ask for a different level by tagging it explicitly. What Ollama does decide automatically is how many layers to put on the GPU, and it will silently fall back to partial CPU offload rather than refuse to load.
# Default tag — Q4_K_M, regardless of your card
ollama pull qwen2.5:7b
# Ask for a level explicitly
ollama pull qwen2.5:7b-instruct-q5_K_M
ollama run qwen2.5:7bCheck it actually ran on the GPU
A model that quietly fell back to CPU offload still answers — it is just slow, and that is the single most common "why is local inference so bad" report. Two checks: `ollama ps` prints a PROCESSOR column that reads 100% GPU when the whole model is on the card, or splits (for example 70%/30% CPU/GPU) when it is not; and `nvidia-smi` should show a process holding roughly the estimate above. If PROCESSOR shows any CPU share, drop to a smaller model or a lower level rather than living with it.
ollama ps
# NAME SIZE PROCESSOR UNTIL
# qwen2.5:7b 5.6 GB 100% GPU 4 minutes from now
nvidia-smi --query-compute-apps=pid,used_memory --format=csvCommon problems
Out of memory on load: the context is usually the cause, not the weights — the KV cache grows linearly with it, so 16K costs a 8B model about 2GB over its 4K figure. Lower the context first. Slow generation with a model that should fit: check `ollama ps` as above; something else on the card (a browser, a game, a second model still resident) is the usual culprit, and `ollama stop <model>` frees the previous one. Windows specifically: the desktop compositor holds VRAM that never appears in your model's own accounting, so treat the 88% band as the real ceiling.
Next steps
Put your own card into the VRAM calculator to see the full list of what fits at the context you actually use — the numbers above are one row of that table. If you are on Windows, the WSL2 + Ollama GPU guide covers driver passthrough, which is where most 8GB setups actually get stuck.
What this guide uses
Next steps
See everything that fits 🟢 RTX 4060Reverse lookup — 8GB at 4096 context, ranked by qualityModels covered in this guide
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
Run GPT-OSS 20B (and 120B) locally without re-quantizing
GPT-OSS ships natively in MXFP4, so the usual "download the Q4_K_M" habit makes it bigger and worse. Sizing, the right flags, and how MoE expert-offload puts the 120B on a 24GB card.
What Can You Run on RTX 4060 Ti 16G?
A practical guide to picking the right model and quant level for NVIDIA's best budget 16GB card.
llama.cpp on Windows with CUDA
Build llama.cpp with NVIDIA GPU support on Windows 11 — the path of least resistance for PC gamers.