The one thing to get right: MXFP4 is the original
Almost every other model on this site is published in BF16 and quantized by the community afterwards, so "find the Q4_K_M" is the right reflex. GPT-OSS breaks that reflex. OpenAI post-trained it with the MoE weights already in MXFP4 (~4.25 bits), and those MoE weights are over 90% of the parameters. The MXFP4 checkpoint is not a lossy copy of something better — it is the model. Converting it up to Q8_0, or sideways to Q4_K_M, gives you a file that is larger and no more accurate, because the precision it is padding back was never there.
gpt-oss-20b MXFP4 (native) ~12.8 GB ← use this
gpt-oss-20b Q8_0 (upcast) ~13.8 GB bigger, not better
gpt-oss-120b MXFP4 (native) ~61 GB ← use this
Rule: for GPT-OSS, "bigger quant" buys you nothing.
Spend the VRAM on context length instead.Sizing it for your card
The 20B fits a 16GB card with room for a useful context window. Note that GPT-OSS uses a head dimension of 64 rather than the usual 128, which halves its KV cache compared to a same-layer-count model — long context is unusually cheap here. Use the VRAM calculator with the MXFP4 level selected; picking Q4_K_M instead will overstate your weights by roughly 14%.
gpt-oss-20b @ MXFP4, batch=1
weights ~12.8 GB
KV cache @ 8K ctx ~0.4 GB
KV cache @ 32K ctx ~1.5 GB
KV cache @ 131K ctx ~6.2 GB
16GB card → comfortable to ~32K ctx
24GB card → full 131K ctx with headroomFastest path: Ollama
Ollama pulls the MXFP4 build by default, so there is no quant tag to choose and no way to accidentally get a re-quantized one. This is the right starting point unless you need custom flags.
ollama pull gpt-oss:20b
ollama run gpt-oss:20b
# 120B — needs ~61GB of combined VRAM+RAM
ollama pull gpt-oss:120b
# OpenAI-compatible endpoint stays on :11434
curl http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"gpt-oss:20b","messages":[{"role":"user","content":"hi"}]}'llama.cpp: pass --jinja or the output gets strange
GPT-OSS was trained on OpenAI's "harmony" response format, which separates the reasoning channel from the final answer. That structure lives in the model's chat template, so llama.cpp needs --jinja to apply it. Skip the flag and you get raw channel markers bleeding into replies, or a model that never stops talking — a failure that reads like a broken quant but is purely a template problem.
# 20B, all layers on a 16GB+ GPU
llama-server \
-hf ggml-org/gpt-oss-20b-GGUF \
--jinja \
-ngl 99 \
--ctx-size 32768 \
--host 0.0.0.0 --port 8080
# --jinja applies the harmony chat template (do not omit)
# -ngl 99 offload every layer to the GPU
# --ctx-size raise freely — KV cache is cheap on this modelRunning the 120B on a 24GB consumer card
This is where the MoE architecture pays off. Only 5.1B parameters are active per token, so the expert weights are read sparsely — which makes them the ideal thing to leave in system RAM. Keep attention and the dense layers on the GPU, push the MoE experts to CPU, and a 61GB model becomes usable on a 24GB card. It is not fast, but it is a genuinely different outcome from "does not fit".
# Offload the MoE experts of N layers to CPU RAM
llama-server \
-hf ggml-org/gpt-oss-120b-GGUF \
--jinja \
-ngl 99 \
--n-cpu-moe 28 \
--ctx-size 16384
# Tune --n-cpu-moe down until you OOM, then back off by 2.
# Lower value = more experts on GPU = faster.
# Needs ~64GB system RAM. Expect single-digit tok/s.Reasoning effort is a dial, not a fixed cost
GPT-OSS exposes low / medium / high reasoning effort. High spends far more tokens thinking before answering, which on local hardware is the difference between a snappy assistant and one that pauses for a minute. Set it low for chat and autocomplete, high only for problems that actually need the chain of thought.
Simplest portable form — put it in the system message:
System: Reasoning: low
Rough local cost on a 16GB card (20B):
low fast, chat-grade latency
medium noticeably more thinking tokens
high can multiply time-to-first-answer several times over
Start at low. Raise it per-task, not globally.Common failure modes
Most GPT-OSS problems reported locally are one of four things, and none of them are the quantization. Check these before hunting for a different build.
Channel markers in the output, or it never stops
→ missing --jinja (harmony template not applied)
"unknown model architecture" on load
→ llama.cpp / Ollama predates gpt-oss support; update
Slower than expected on the 120B
→ --n-cpu-moe too high; lower it until GPU VRAM is nearly full
File is much bigger than ~12.8GB (20B)
→ you downloaded an upcast build; get the MXFP4 oneWhat this guide uses
- Hardware
- RTX 409024GBRTX 3090
- Format
- GGUF
- Picks for it
- Best local LLM for 24GB
Next steps
See everything that fits 🟢 RTX 4090Reverse lookup — 24GB at 32768 context, ranked by qualityModels covered in this guide
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
llama.cpp on Windows with CUDA
Build llama.cpp with NVIDIA GPU support on Windows 11 — the path of least resistance for PC gamers.
Quantize Your Own Model to GGUF
Use llama.cpp's quantize tool to convert any HF model to GGUF Q4_K_M for local inference.
8GB GPU Starter Guide: 3060 / 4060 / 3070
The most common local LLM hardware tier — which models, quants, and context lengths actually fit in 8GB VRAM.