ggml_cuda_init: failed to initialize CUDA

The message

ggml_cuda_init: failed to initialize CUDA: <reason>

Parts in <angle brackets> change from run to run.

What it means

A CUDA build of llama.cpp started but could not talk to any NVIDIA GPU, so it falls back to the CPU. The text after the colon is CUDA's own reason.

Why it happens

• Inside Docker: the container was started without GPU access (--gpus all), or the NVIDIA Container Toolkit is missing.
• The installed driver is older than the CUDA version the binary was built with.
• In WSL2: a Linux NVIDIA driver was installed inside WSL over the Windows passthrough driver.

How to fix it

1. Run nvidia-smi where llama.cpp runs (host, container or WSL). If it fails there too, fix the driver first.
2. Update the NVIDIA driver; prebuilt CUDA binaries name the CUDA version they need.
3. In WSL2 install only the CUDA toolkit inside Linux (cuda-toolkit-12-x), never the cuda or cuda-drivers packages.

Guides that cover this

• WSL2 + Ollama GPU Passthrough on Windows
• Docker: Ollama with NVIDIA or AMD GPU Passthrough
• llama.cpp on Windows with CUDA

Related errors

• Docker: could not select device driver with [[gpu]]
• Ollama shows 100% CPU instead of the GPU