The one thing to get right: MXFP4 is the original
Almost every other model on this site is published in BF16 and quantized by the community afterwards, so "find the Q4_K_M" is the right reflex. GPT-OSS breaks that reflex. OpenAI post-trained it with the MoE weights already in MXFP4 (~4.25 bits), and those MoE weights are over 90% of the parameters. The MXFP4 checkpoint is not a lossy copy of something better — it is the model. Converting it up to Q8_0, or sideways to Q4_K_M, gives you a file that is larger and no more accurate, because the precision it is padding back was never there.
gpt-oss-20b MXFP4 (native) ~12.8 GB ← use this
gpt-oss-20b Q8_0 (upcast) ~13.8 GB bigger, not better
gpt-oss-120b MXFP4 (native) ~61 GB ← use this
Rule: for GPT-OSS, "bigger quant" buys you nothing.
Spend the VRAM on context length instead.Sizing it for your card
The 20B fits a 16GB card with room for a useful context window. Note that GPT-OSS uses a head dimension of 64 rather than the usual 128, which halves its KV cache compared to a same-layer-count model — long context is unusually cheap here. Use the VRAM calculator with the MXFP4 level selected; picking Q4_K_M instead will overstate your weights by roughly 14%.
gpt-oss-20b @ MXFP4, batch=1
weights ~12.8 GB
KV cache @ 8K ctx ~0.4 GB
KV cache @ 32K ctx ~1.5 GB
KV cache @ 131K ctx ~6.2 GB
16GB card → comfortable to ~32K ctx
24GB card → full 131K ctx with headroomFastest path: Ollama
Ollama pulls the MXFP4 build by default, so there is no quant tag to choose and no way to accidentally get a re-quantized one. This is the right starting point unless you need custom flags.
ollama pull gpt-oss:20b
ollama run gpt-oss:20b
# 120B — needs ~61GB of combined VRAM+RAM
ollama pull gpt-oss:120b
# OpenAI-compatible endpoint stays on :11434
curl http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"gpt-oss:20b","messages":[{"role":"user","content":"hi"}]}'llama.cpp: pass --jinja or the output gets strange
GPT-OSS was trained on OpenAI's "harmony" response format, which separates the reasoning channel from the final answer. That structure lives in the model's chat template, so llama.cpp needs --jinja to apply it. Skip the flag and you get raw channel markers bleeding into replies, or a model that never stops talking — a failure that reads like a broken quant but is purely a template problem.
# 20B, all layers on a 16GB+ GPU
llama-server \
-hf ggml-org/gpt-oss-20b-GGUF \
--jinja \
-ngl 99 \
--ctx-size 32768 \
--host 0.0.0.0 --port 8080
# --jinja applies the harmony chat template (do not omit)
# -ngl 99 offload every layer to the GPU
# --ctx-size raise freely — KV cache is cheap on this modelRunning the 120B on a 24GB consumer card
This is where the MoE architecture pays off. Only 5.1B parameters are active per token, so the expert weights are read sparsely — which makes them the ideal thing to leave in system RAM. Keep attention and the dense layers on the GPU, push the MoE experts to CPU, and a 61GB model becomes usable on a 24GB card. It is not fast, but it is a genuinely different outcome from "does not fit".
# Offload the MoE experts of N layers to CPU RAM
llama-server \
-hf ggml-org/gpt-oss-120b-GGUF \
--jinja \
-ngl 99 \
--n-cpu-moe 28 \
--ctx-size 16384
# Tune --n-cpu-moe down until you OOM, then back off by 2.
# Lower value = more experts on GPU = faster.
# Needs ~64GB system RAM. Expect single-digit tok/s.Reasoning effort is a dial, not a fixed cost
GPT-OSS exposes low / medium / high reasoning effort. High spends far more tokens thinking before answering, which on local hardware is the difference between a snappy assistant and one that pauses for a minute. Set it low for chat and autocomplete, high only for problems that actually need the chain of thought.
Simplest portable form — put it in the system message:
System: Reasoning: low
Rough local cost on a 16GB card (20B):
low fast, chat-grade latency
medium noticeably more thinking tokens
high can multiply time-to-first-answer several times over
Start at low. Raise it per-task, not globally.Common failure modes
Most GPT-OSS problems reported locally are one of four things, and none of them are the quantization. Check these before hunting for a different build.
Channel markers in the output, or it never stops
→ missing --jinja (harmony template not applied)
"unknown model architecture" on load
→ llama.cpp / Ollama predates gpt-oss support; update
Slower than expected on the 120B
→ --n-cpu-moe too high; lower it until GPU VRAM is nearly full
File is much bigger than ~12.8GB (20B)
→ you downloaded an upcast build; get the MXFP4 oneRelated guides
llama.cpp on Windows with CUDA
Build llama.cpp with NVIDIA GPU support on Windows 11 — the path of least resistance for PC gamers.
Quantize Your Own Model to GGUF
Use llama.cpp's quantize tool to convert any HF model to GGUF Q4_K_M for local inference.
8GB GPU Starter Guide: 3060 / 4060 / 3070
The most common local LLM hardware tier — which models, quants, and context lengths actually fit in 8GB VRAM.