What you need first
Visual Studio 2022 with the "Desktop development with C++" workload — that one checkbox also installs CMake and the MSVC toolchain, so there is nothing else to fetch. The CUDA Toolkit, matching a driver new enough for it. And the habit that saves the most time: every command below runs in a Developer Command Prompt for VS 2022, not a plain terminal, because a plain terminal has none of the compiler paths set.
# In a Developer Command Prompt / PowerShell for VS 2022
cmake --version
nvcc --version
nvidia-smiBuild with CUDA on
The flag is `GGML_CUDA`, not `LLAMA_CUDA`. This matters more than it looks: CMake ignores an unknown `-D` without complaining, so the old name produces a clean, successful, CPU-only build — and the first sign of trouble is a model that runs at a tenth of the speed you expected.
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
# CMAKE_CUDA_ARCHITECTURES is optional but cuts build time a lot:
# 86 = RTX 30-series, 89 = RTX 40-series, 120 = RTX 50-series
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="89"
cmake --build build --config Release -jRun the server
The binaries land in `build\bin\Release\` and are named `llama-server.exe` and `llama-cli.exe` — the old `server.exe` and `main.exe` names are gone, which is the other thing that sends people to a search engine. Bind to localhost unless you actually intend to expose the machine.
.\build\bin\Release\llama-server.exe ^
--model C:\models\Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf ^
--n-gpu-layers 99 ^
--ctx-size 4096 ^
--host 127.0.0.1 --port 8080Check it actually ran on the GPU
llama.cpp prints its layer placement at load time, and that line is the whole answer — if the offload count is lower than the layer count, the rest is on your CPU and every token pays for it. A CPU-only build reports no CUDA device at all, which is how you catch the wrong-flag build described above.
# What a working CUDA build prints:
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 4060 Ti, compute capability 8.9
load_tensors: offloaded 33/33 layers to GPU
# A CPU-only build never mentions a CUDA device at all.What the numbers should look like
Token generation reads the whole weight set once per token, so your card’s memory bandwidth is the ceiling. Llama 3.1 8B at Q4_K_M is 4.6 GB of weights: on a 288 GB/s RTX 4060 Ti that caps generation near 62 tok/s, on a 1,008 GB/s RTX 4090 near 218. Those are arithmetic on published specifications rather than benchmarks, and real throughput lands below them — but an order of magnitude below means you are on the CPU.
When it does not work
`nvcc` not found: you are not in a Developer Command Prompt, or the CUDA Toolkit went in after Visual Studio and its MSBuild integration never registered. The build succeeds but there is no CUDA device: you passed `LLAMA_CUDA` instead of `GGML_CUDA`, and CMake ignored it — delete `build\` and configure again, because a stale cache keeps the old answer. Everything is mysteriously slow rather than failing: Windows has System Memory Fallback on by default in the NVIDIA control panel, which spills VRAM into system RAM instead of reporting an out-of-memory error, so a model that does not fit becomes a model that crawls. Turn it off while you are measuring. And an unspecified compiler error deep in a CUDA header usually means the toolkit and the Visual Studio version disagree — check the toolkit’s supported MSVC range before suspecting your code.
Common questions
Why does my llama.cpp build ignore the GPU on Windows?
Almost always the build flag. It is `-DGGML_CUDA=ON`; the older `LLAMA_CUDA` name no longer does anything, and CMake ignores an unknown `-D` silently, so you get a successful CPU-only build with no warning. Delete the `build` directory before reconfiguring — a stale CMake cache will keep the previous answer. A working build prints its CUDA device and its layer offload count at load time.
Do I need WSL to run llama.cpp with CUDA on Windows?
No. llama.cpp builds natively with MSVC and the CUDA Toolkit, and the resulting `llama-server.exe` uses the GPU directly. WSL is a reasonable choice if you want a Linux toolchain for other reasons, but it adds a layer rather than removing one, and native builds avoid the filesystem performance question entirely.
What is System Memory Fallback and should I turn it off?
It is an NVIDIA driver feature on Windows that spills VRAM into system RAM when a workload does not fit, instead of failing with an out-of-memory error. For gaming that is a kindness; for local inference it turns a clear error into a model that runs at a fraction of its speed for no visible reason. Turn it off while you are sizing models, so that "does not fit" fails loudly.
What this guide uses
- Format
- GGUF
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
Quantize Your Own Model to GGUF
Use llama.cpp's quantize tool to convert any HF model to GGUF Q4_K_M for local inference.
WSL2 + Ollama GPU Passthrough on Windows
Run Ollama with NVIDIA GPU acceleration inside WSL2 — the most reliable Windows path for local LLMs.
Run GPT-OSS 20B (and 120B) locally without re-quantizing
GPT-OSS ships natively in MXFP4, so the usual "download the Q4_K_M" habit makes it bigger and worse. Sizing, the right flags, and how MoE expert-offload puts the 120B on a 24GB card.