Skip to content

Can RTX 4090 run Llama 3.1 8B Instruct?

Comfortable: GGUF Q8_0 needs 9.5 GB at 4K, batch 1; headroom 14.5 GB.

Text input, FP16 KV cache, one card. Unified-memory capacity uses the GPU guide’s usable limit; negative headroom may require host offload. Speed source settings may differ from this 4K estimate.

Measured runs

FrameworkQuanttok/sVRAM used (GB)
ExLlamaV2EXL2 4.65bpw2355.4
vLLMAWQ INT42184.9
llama.cppGGUF Q4_K_M1485.7

Quant memory at 4K

QuantTotal GBVerdictWeight sizing
GGUF Q2_K3.3ComfortableEstimated weights
GGUF Q4_K_M5.6ComfortableEstimated weights
GGUF Q6_K7.4ComfortableEstimated weights
GGUF Q8_09.5ComfortableEstimated weights
AWQ INT44.8ComfortableEstimated weights
EXL2 4.65bpw5.4ComfortableEstimated weights

Context capacity · GGUF Q8_0

ContextTotal GBVerdict
4K9.5Comfortable
16K11.1Comfortable
32K13.3Comfortable
128K26.5Does not fit

Generation speed

Measured: 235 tok/s, EXL2 4.65bpw, ExLlamaV2

Measured: 218 tok/s, AWQ INT4, vLLM

Measured: 148 tok/s, GGUF Q4_K_M, llama.cpp

GGUF Q8_0: 83–109 tok/s · Derived range

Other model / GPU pairs