Skip to content

Can RTX 4090 run Qwen2.5 32B Instruct?

Comfortable: EXL2 3.5bpw needs 16.0 GB at 4K, batch 1; headroom 8.0 GB.

Text input, FP16 KV cache, one card. Unified-memory capacity uses the GPU guide’s usable limit; negative headroom may require host offload. Speed source settings may differ from this 4K estimate.

Measured runs

FrameworkQuanttok/sVRAM used (GB)
llama.cppGGUF Q4_K_M4422

Quant memory at 4K

QuantTotal GBVerdictWeight sizing
GGUF Q3_K_M17.5ComfortableEstimated weights
GGUF Q4_K_M21.7TightEstimated weights
EXL2 3.5bpw16.0ComfortableEstimated weights

Context capacity · EXL2 3.5bpw

ContextTotal GBVerdict
4K16.0Comfortable
16K19.3Comfortable
32K23.7Tight
128K50.1Does not fit

Generation speed

Measured: 44 tok/s, GGUF Q4_K_M, llama.cpp

EXL2 3.5bpw: 75 tok/s · Bandwidth ceiling (not measured)

Other model / GPU pairs