What you need first
A 16GB RTX 4060 Ti — not the 8GB card of the same name, which is a different machine for this purpose. A recent NVIDIA driver, and either Ollama (simplest) or a CUDA build of llama.cpp. Nothing here needs more than 16GB of system RAM, because the weights live on the card.
# Confirm you have the 16GB card, not the 8GB one
nvidia-smi --query-gpu=name,memory.total --format=csvThe honest summary of this card
Capacity is generous and bandwidth is not. 52 of the models in this index fit it comfortably at 4K context and 61 load at all, which is the same list a 16GB RTX 4080 Super returns — but the 4060 Ti reads its weights at 288 GB/s against that card’s 736. Generating a token means reading every weight once, so the ceiling on throughput is roughly a third of what the same file does on the faster card. Pick models expecting that, not the capacity.
The three worth starting with
Qwen3 14B at Q4_K_M needs 10.0 GB at 4K context and gives up 2.6% perplexity against FP16 — the best general-purpose fit, with 6 GB spare for a longer window. GPT-OSS 20B needs 11.4 GB: it is a mixture-of-experts model, so it reads only a fraction of its weights per token and runs faster than its size suggests, and its 4-bit weights are the released checkpoint rather than a conversion. Mistral Small 24B at AWQ INT4 is the largest thing that fits at all, at 13.2 GB — 82% of the card, with nothing left for context.
ollama pull qwen3:14b
ollama run qwen3:14bCheck it actually ran on the GPU
This is the step people skip. Both Ollama and llama.cpp fall back to the CPU silently when a layer will not fit, and the only symptom is that everything is slow. Watch the card while a prompt is generating: memory used should jump by roughly the model size, and utilisation should be high rather than near zero.
# While a prompt is generating, in a second terminal:
nvidia-smi --query-gpu=memory.used,utilization.gpu --format=csv -l 1
# llama.cpp prints the split at load time — every layer should be on the GPU:
# load_tensors: offloaded 41/41 layers to GPUWhat the numbers should look like
At 288 GB/s, Qwen3 14B at Q4_K_M is 8.5 GB of weights, which puts a hard ceiling near 34 tok/s — arithmetic on two published numbers, not a benchmark, and real throughput lands below it. If you are seeing single digits on a model that fits, you are on the CPU. Memory is the other number to watch: Qwen3 14B goes from 10.0 GB at 4K to 12.1 GB at 16K and 14.9 GB at 32K. The weights do not change; the KV cache is what grows.
When it does not work
Out of memory at a context length that used to work: the KV cache grew, not the model — drop the window or the quant level. Everything is slow and nvidia-smi shows low utilisation: layers went to the CPU, so lower the offload count until it fits entirely or pick a smaller quant. A 30B model will not load: Qwen3 30B-A3B needs 19.7 GB, which is 3.7 GB over this card — offloading part of it to system RAM will load it at a speed set by your DIMMs, which is a different machine rather than a smaller quant. And on Windows, the NVIDIA control panel’s System Memory Fallback silently spills VRAM into system RAM instead of failing: that turns an out-of-memory error into a mysteriously slow model, so turn it off while you are measuring.
Common questions
Is the 16GB RTX 4060 Ti good for local LLMs?
For capacity, yes — 52 of the models in this index fit it comfortably at 4K context, including 14B models at Q4_K_M with room to spare. For speed, it is the slowest 16GB card here: 288 GB/s against 736 on an RTX 4080 Super holding exactly the same models. It is a good card for running larger models slowly and a poor one for running small models fast.
Can an RTX 4060 Ti 16G run a 30B model?
Not entirely on the card. Qwen3 30B-A3B at Q4_K_M needs about 19.7 GB at 4K context, 3.7 GB more than the card has. The largest model that does clear it comfortably is Mistral Small 24B at AWQ INT4, at 13.2 GB. Splitting a model between VRAM and system RAM works, at a speed set by the slower half.
How many tokens per second should I expect on a 14B model?
This index has no measured run on this card, so it does not publish a figure. What the specification allows: Qwen3 14B at Q4_K_M is 8.5 GB of weights and the card moves 288 GB/s, so generation cannot exceed roughly 34 tok/s and will land below it. Single-digit throughput on a model that fits means layers are on the CPU.
What this guide uses
- Picks for it
- Best local LLM for 16GB
Next steps
See everything that fits 🟢 RTX 4060 Ti 16GReverse lookup — 16GB at 8192 context, ranked by qualityModels covered in this guide
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
DeepSeek-R1 Distill 14B: EXL2 vs GGUF
Head-to-head on RTX 4090 — when to pick turboderp EXL2 over bartowski GGUF.
llama.cpp on Windows with CUDA
Build llama.cpp with NVIDIA GPU support on Windows 11 — the path of least resistance for PC gamers.
8GB GPU Starter Guide: 3060 / 4060 / 3070
The most common local LLM hardware tier — which models, quants, and context lengths actually fit in 8GB VRAM.