Prerequisites
Install Visual Studio Build Tools, CMake, and CUDA Toolkit 12.x.
powershell
winget install Kitware.CMake
# CUDA: download from developer.nvidia.com/cuda-downloadsBuild
Enable CUDA backend during CMake configure.
powershell
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DLLAMA_CUDA=ON
cmake --build build --config Release -jRelated guides
AdvancedEdge / Local12 min read
Quantize Your Own Model to GGUF
Use llama.cpp's quantize tool to convert any HF model to GGUF Q4_K_M for local inference.
IntermediateEdge / Local10 min read
WSL2 + Ollama GPU Passthrough on Windows
Run Ollama with NVIDIA GPU acceleration inside WSL2 — the most reliable Windows path for local LLMs.
IntermediateEdge / Local9 min read
Run GPT-OSS 20B (and 120B) locally without re-quantizing
GPT-OSS ships natively in MXFP4, so the usual "download the Q4_K_M" habit makes it bigger and worse. Sizing, the right flags, and how MoE expert-offload puts the 120B on a 24GB card.
Deployment guides are educational. Each model is subject to its own license — read the official Hugging Face model card before downloading or deploying.