IntermediateEdge / Local 8 min read

Qwen2.5-Coder 32B on a Single RTX 4090

The best open coding model that fits in 24GB — quant selection and tuning tips.

Verified stack

RTX 4090 24G · Qwen2.5-Coder 32B Q4_K_M · llama.cpp -ngl 99 · ctx 4K

Last verified 2026-07-22

Qwen2.5-Coder32BRTX 4090GGUF

Quant choice

Q4_K_M uses ~22GB for weights alone. Drop to Q3_K_M or EXL2 3.5bpw if you need 8K+ context.

text

Q4_K_M: best quality, ~22 GB weights
Q3_K_M: saves ~4 GB, acceptable for code
EXL2 3.5bpw: fastest, ~16 GB weights

Recommended command

llama.cpp with full GPU offload and 4K context is the sweet spot.

bash

./build/bin/llama-server \
  -m ./models/Qwen2.5-Coder-32B-Q4_K_M.gguf \
  -ngl 99 -c 4096 --host 0.0.0.0 --port 8080

Related guides

IntermediateEdge / Local9 min read

DeepSeek-R1 Distill 14B: EXL2 vs GGUF

Head-to-head on RTX 4090 — when to pick turboderp EXL2 over bartowski GGUF.

IntermediateEdge / Local10 min read

ExLlamaV2 on RTX 4090: Full Setup Guide

Install ExLlamaV2, load an EXL2 quant, and serve an OpenAI-compatible API in under 10 minutes.

IntermediateEdge / Local9 min read

Run GPT-OSS 20B (and 120B) locally without re-quantizing

GPT-OSS ships natively in MXFP4, so the usual "download the Q4_K_M" habit makes it bigger and worse. Sizing, the right flags, and how MoE expert-offload puts the 120B on a 24GB card.

Deployment guides are educational. Each model is subject to its own license — read the official Hugging Face model card before downloading or deploying.