What you need first
A fine-tuned or merged model in Hugging Face safetensors format, a built copy of llama.cpp, and enough free disk for three copies of the model at once: the original weights, an intermediate high-precision GGUF, and the final quantized file. For a 7B model in BF16 that is roughly 14 GB + 14 GB + 5 GB — budget 35–40 GB of scratch space, not just the size of the model you started with.
pip install -r requirements.txt # inside your llama.cpp checkout
df -h .Convert to GGUF
The conversion script is `convert_hf_to_gguf.py`. The older `convert.py` name that circulates in older tutorials no longer exists in the repository — if a guide you are reading elsewhere still uses it, that guide predates the rename. Convert to a high-precision intermediate first; converting straight to a low bit-depth from safetensors is not how the pipeline works.
python convert_hf_to_gguf.py /path/to/my-model \
--outfile my-model-f16.gguf \
--outtype f16Quantize it
The quantization binary is `llama-quantize` — the bare `quantize` name from older tutorials was renamed along with the rest of the CLI tools. Q4_K_M is the level almost everyone should start with: this index’s own median published quality loss at that level is 2.9% against FP16, and every model in this index that ships GGUF ships this level.
./build/bin/llama-quantize my-model-f16.gguf my-model-Q4_K_M.gguf Q4_K_MDo it properly: an importance matrix
A plain quantization treats every weight the same. An importance matrix (imatrix), computed by running representative text through the F16 model first, tells the quantizer which weights matter more and protects them — the same idea behind why Q4_K_M already beats a naive 4-bit round-off, taken further. This step is what separates a quant you would publish from one that is merely usable.
./build/bin/llama-imatrix \
-m my-model-f16.gguf \
-f calibration-data.txt \
-o my-model.imatrix
./build/bin/llama-quantize \
--imatrix my-model.imatrix \
my-model-f16.gguf my-model-Q4_K_M.gguf Q4_K_MCheck it actually worked
Load the quantized file and run a real prompt — a successful conversion that produces garbage output is a real failure mode, usually from a tokenizer or chat-template mismatch during conversion rather than from quantization itself. Compare the file size against expectation: a Q4_K_M file should land near params × 0.6 bytes per parameter, and a figure wildly off that suggests the conversion picked up the wrong precision.
./build/bin/llama-cli -m my-model-Q4_K_M.gguf -p "Explain quantization in one sentence." -n 64
ls -lh my-model-Q4_K_M.gguf # sanity-check the sizeWhen it does not work
The conversion script fails with an unknown architecture: your model uses a layer type `convert_hf_to_gguf.py` does not yet recognise — check the script’s supported-architecture list before assuming your model is fine and the tool is broken. Output is fluent but wrong (repeats itself, ignores the prompt): usually a chat-template mismatch carried over from the original model, not a quantization artifact — verify with the F16 intermediate before blaming the quant step. Running out of disk mid-conversion: this is the three-copies problem from the first section: clean up the F16 intermediate once the quantized file is verified, but not before. And if you would rather not run any of this yourself, the GGUF-my-repo Space on Hugging Face runs the same pipeline and syncs from llama.cpp’s main branch every six hours.
Common questions
What is convert_hf_to_gguf.py, and where did convert.py go?
`convert_hf_to_gguf.py` is the current script for turning Hugging Face safetensors weights into a GGUF file. `convert.py` was the older name and no longer exists in the repository — any tutorial still referencing it predates the rename.
What is llama-quantize, and is it the same as quantize?
It is the same tool under its current name. The binary was renamed from the bare `quantize` along with the rest of llama.cpp’s CLI tools (`llama-server`, `llama-cli`), so an older tutorial referencing `./quantize` is naming a file that no longer exists in a fresh build.
Do I need an importance matrix (imatrix) to quantize a model?
No — quantization works without one. An imatrix improves the result by telling the quantizer which weights matter most, computed by running representative text through the F16 model first. Skip it for a quick personal test; use it before publishing a quant anyone else will rely on.
What this guide uses
- Format
- GGUF
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
llama.cpp on Windows with CUDA
Build llama.cpp with NVIDIA GPU support on Windows 11 — the path of least resistance for PC gamers.
Run GPT-OSS 20B (and 120B) locally without re-quantizing
GPT-OSS ships natively in MXFP4, so the usual "download the Q4_K_M" habit makes it bigger and worse. Sizing, the right flags, and how MoE expert-offload puts the 120B on a 24GB card.
AMD GPU + llama.cpp via ROCm (Quick Start)
Run GGUF models on Radeon RX 7900 / 6800 series with llama.cpp HIP backend — what works and what does not.