Who this is for
A Windows machine with an NVIDIA GPU, where you want the Linux tooling (Ollama, llama.cpp, Python) without dual-booting. If you only want to chat with a model and never touch a terminal, the native Windows Ollama app is simpler — this guide is for the case where you also want the Linux side.
Prerequisites
Windows 11 (or Windows 10 21H2+), an NVIDIA GPU, and a current NVIDIA driver installed on Windows. The single most important rule: do not install an NVIDIA driver inside WSL. The Windows driver projects CUDA into the WSL kernel through /usr/lib/wsl/lib, and installing a Linux driver on top overwrites those stubs and breaks passthrough — this is the most common way a working setup stops working.
# PowerShell (Administrator)
wsl --install
wsl --update
wsl --status # want: default version 2
# The check that matters — GPU visible from inside the WSL VM
wsl nvidia-smiGive the WSL VM enough RAM
WSL2 runs in a lightweight VM with its own memory limit, historically about half of host RAM. That ceiling is invisible until a model needs CPU offload and the VM runs out well before Windows does. Set it explicitly in %UserProfile%\.wslconfig, then wsl --shutdown to apply. Leave several GB for Windows itself.
# %UserProfile%\.wslconfig (example for a 32GB machine)
[wsl2]
memory=20GB
swap=8GB
# then, in PowerShell:
# wsl --shutdownInstall Ollama inside WSL
Install the Linux build inside Ubuntu, not the Windows app — running both leaves two servers competing for port 11434 and for the GPU. Keep models on the WSL filesystem (~/.ollama); putting them under /mnt/c crosses the 9p filesystem boundary on every read and is dramatically slower to load.
curl -fsSL https://ollama.com/install.sh | sh
# Default tag is Q4_K_M for most models, regardless of your card
ollama pull qwen2.5:7b
ollama run qwen2.5:7bVerify it is actually on the GPU
Ollama will fall back to CPU rather than fail, so a model that answers slowly is the symptom of a broken passthrough, not of a slow card. Two checks: ollama ps shows a PROCESSOR column that reads 100% GPU when the whole model is resident on the card, and nvidia-smi inside WSL should show the ollama process holding roughly the model size. If PROCESSOR shows a CPU share on a model that should fit, the passthrough is the problem, not the model.
ollama ps
# NAME SIZE PROCESSOR UNTIL
# qwen2.5:7b 5.1 GB 100% GPU 4 minutes from now
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csvReaching the API from Windows
WSL2 forwards localhost, so http://localhost:11434 from a Windows browser or PowerShell reaches the server inside WSL with no extra configuration. Prefer that over binding Ollama to 0.0.0.0: the WSL VM sits on a bridged network, and a server bound to all interfaces there is reachable from your LAN with no authentication in front of it.
# From Windows PowerShell
curl http://localhost:11434/api/tagsCommon problems
wsl nvidia-smi fails: update the Windows driver, then wsl --update and wsl --shutdown; do not install a driver inside WSL. nvidia-smi works but Ollama uses CPU: usually a second Ollama (the Windows app) already holding the port, or a model too large for the card — check ollama ps. Disk fills up: the WSL virtual disk grows to hold pulled models and does not shrink on its own; ollama rm removes a model, and reclaiming the space needs a manual compact of the vhdx. First load painfully slow: the model is probably under /mnt/c.
Next steps
Put your card into the VRAM calculator to see what else fits at the context you actually use before pulling a larger model — on Windows, subtract the 0.5–1.5GB the desktop already holds.
What this guide uses
- Format
- GGUF
- Picks for it
- Best local LLM for 16GB
Next steps
See everything that fits 🟢 RTX 4060 Ti 16GReverse lookup — 16GB at 8192 context, ranked by qualityModels covered in this guide
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
Ollama on Windows (Native, No WSL)
Install the Windows Ollama app for the simplest path — GPU works on NVIDIA; AMD is CPU-only for now.
llama.cpp on Windows with CUDA
Build llama.cpp with NVIDIA GPU support on Windows 11 — the path of least resistance for PC gamers.
Run GPT-OSS 20B (and 120B) locally without re-quantizing
GPT-OSS ships natively in MXFP4, so the usual "download the Q4_K_M" habit makes it bigger and worse. Sizing, the right flags, and how MoE expert-offload puts the 120B on a 24GB card.