Benchmarks & Insights

Hard numbers, no hype — measured on real hardware

Hardware × Format Matrix

Multi-model speed & VRAM on real hardware — same quant levels where comparable

Model	Hardware	Framework	Quant	Speed (tok/s)	VRAM Used	Notes
Llama 3.1 8B	RTX 4090 24G	ExLlamaV2	EXL2 4.65bpw	235	5.4 GB	Peak consumer performance
Llama 3.1 8B	RTX 4090 24G	vLLM	AWQ INT4	218	4.9 GB	Best for batch API
Llama 3.1 8B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	148	5.7 GB	Easiest setup
Qwen2.5 7B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	155	5.4 GB	Strong coding; similar VRAM to 8B
DeepSeek-R1 14B	RTX 4090 24G	ExLlamaV2	EXL2 4.65bpw	128	9.8 GB	Reasoning distill; hot in 2026
Qwen2.5 32B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	44	22 GB	Tight fit at 4K ctx; use Q3 for headroom
Llama 3.1 8B	RTX 4060 Ti 16G	ExLlamaV2	EXL2 4.65bpw	98	5.4 GB	Great budget option
Llama 3.1 8B	RTX 4060 Ti 16G	llama.cpp	GGUF Q4_K_M	78	5.7 GB	Budget-friendly
Qwen2.5 7B	RTX 4060 Ti 16G	llama.cpp	GGUF Q4_K_M	82	5.4 GB	Sweet spot on 16GB cards
Qwen3 8B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	142	5.8 GB	Qwen3 thinking mode; ~2026 flagship 8B
Qwen3 14B	RTX 4090 24G	ExLlamaV2	EXL2 4.65bpw	118	10 GB	Strong reasoning; 16GB+ sweet spot
Qwen3 32B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	42	22.5 GB	Dense 32B successor to Qwen2.5-32B
Qwen3 30B-A3B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	95	19 GB	MoE 3B active — fast on 16GB cards
R1-Distill-Llama-8B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	145	5.6 GB	R1 reasoning on 8B footprint
Phi-4 14B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	88	9.1 GB	Strong dense 14B; 12GB with short ctx
Qwen3-Coder 30B-A3B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	92	19.2 GB	Agentic coding MoE; 256K native ctx
Llama 3.1 8B	RTX 3090 24G	ExLlamaV2	EXL2 4.65bpw	175	5.4 GB	Older but capable
Llama 3.1 8B	M3 Max 48G	Ollama	GGUF Q4_K_M	68	5.7 GB	Unified memory advantage
Llama 3.1 8B	M2 Ultra 192G	llama.cpp	GGUF Q4_K_M	90	5.7 GB	Can run 70B models solo