AdvancedEdge / Local 12 min read

Quantize Your Own Model to GGUF

Use llama.cpp's quantize tool to convert any HF model to GGUF Q4_K_M for local inference.

Verified stack

llama.cpp convert_hf_to_gguf.py · llama-quantize · Q4_K_M target

Last verified 2026-07-22

GGUFllama.cppquantizecustom

Convert HF to GGUF

First convert the safetensors model to FP16 GGUF, then quantize.

bash

python convert_hf_to_gguf.py ./my-model --outfile my-model-f16.gguf
./build/bin/llama-quantize my-model-f16.gguf my-model-Q4_K_M.gguf Q4_K_M

Related guides

BeginnerEdge / Local9 min read

llama.cpp on Windows with CUDA

Build llama.cpp with NVIDIA GPU support on Windows 11 — the path of least resistance for PC gamers.

IntermediateEdge / Local9 min read

Run GPT-OSS 20B (and 120B) locally without re-quantizing

GPT-OSS ships natively in MXFP4, so the usual "download the Q4_K_M" habit makes it bigger and worse. Sizing, the right flags, and how MoE expert-offload puts the 120B on a 24GB card.

AdvancedEdge / Local11 min read

Running 70B on Dual RTX 3090 with llama.cpp

Tensor-split across two 24GB cards to run Llama 3.1 70B or Qwen2.5 72B at Q4.

Deployment guides are educational. Each model is subject to its own license — read the official Hugging Face model card before downloading or deploying.