Benchmarks & Insights

Hard numbers, no hype — measured on real hardware

Every row below was produced on a machine described on this page, not collected from elsewhere. That also means the coverage is narrow: 16 runs across 4 cards, out of 63 in the GPU index. Where a figure on this site is not measured, it says estimated — this page is the only place the measured ones come from.

The same figures, as text

Charts are drawn as SVG paths, so the numbers in them are not readable text. These tables carry the same data.

What is and is not covered

Measured here

Estimated only

The other 59 cards in the index. Their numbers come from the same formula the VRAM calculator uses, scaled by published memory bandwidth — not a second, hidden measurement.

Not covered

Multi-GPU throughput beyond the dual-RTX-3090 guide
Batch sizes above 1
Prompt processing (prefill) speed, as distinct from generation
Quality on anything other than perplexity

Hardware × Format Matrix

Multi-model speed & VRAM on real hardware — same quant levels where comparable

Hardware × Format Matrix — Multi-model speed & VRAM on real hardware — same quant levels where comparable
Model	Hardware	Framework	Quant	Speed (tok/s)	VRAM Used	Notes
Llama 3.1 8B	RTX 4090 24G	ExLlamaV2	EXL2 4.65bpw	235	5.4 GB	Peak consumer performance
Llama 3.1 8B	RTX 4090 24G	vLLM	AWQ INT4	218	4.9 GB	Best for batch API
Llama 3.1 8B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	148	5.7 GB	Easiest setup
Qwen2.5 7B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	155	5.4 GB	Strong coding; similar VRAM to 8B
DeepSeek-R1 14B	RTX 4090 24G	ExLlamaV2	EXL2 4.65bpw	128	9.8 GB	Reasoning distill; hot in 2026
Qwen2.5 32B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	44	22 GB	Tight fit at 4K ctx; use Q3 for headroom
Qwen3 8B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	142	5.8 GB	Qwen3 thinking mode; ~2026 flagship 8B
Qwen3 14B	RTX 4090 24G	ExLlamaV2	EXL2 4.65bpw	118	10 GB	Strong reasoning; 16GB+ sweet spot
Qwen3 32B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	42	22.5 GB	Dense 32B successor to Qwen2.5-32B
Qwen3 30B-A3B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	95	19 GB	MoE 3B active — fast on 16GB cards
R1-Distill-Llama-8B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	145	5.6 GB	R1 reasoning on 8B footprint
Phi-4 14B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	88	9.1 GB	Strong dense 14B; 12GB with short ctx
Qwen3-Coder 30B-A3B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	92	19.2 GB	Agentic coding MoE; 256K native ctx
Llama 3.1 8B	RTX 3090 24G	ExLlamaV2	EXL2 4.65bpw	175	5.4 GB	Older but capable
Llama 3.1 8B	M3 Max 48G	Ollama	GGUF Q4_K_M	68	5.7 GB	Unified memory advantage
Llama 3.1 8B	M2 Ultra 192G	llama.cpp	GGUF Q4_K_M	90	5.7 GB	Can run 70B models solo

Reference Model

Meta Llama 3.1 8B Instruct

PPL Dataset

WikiText-2

Context

4096 tokens

Batch Size

Framework Versions

llamacpp: b4217 (CUDA backend)exllama: ExLlamaV2 0.2.1vllm: v0.6.3ollama: 0.3.14

Current releases of those runtimes (checked 2026-09-15)

llamacpp: b10978vllm: v0.29.0ollama: v0.34.1exllama: v0.3.2 (stalled)

The figures on this page were measured on the stack in the left column. Those releases are now well behind current — speed numbers in particular move with the runtime, so treat them as a ranking between formats rather than as what you will see today. One thing this comparison cannot show: ExLlamaV2 has had no release since 2025-07, and the maintainer's active project has moved to a separate repository, ExLlamaV3 (v1.5.0 and shipping fast). Its own README describes a genuinely new quantization format, EXL3, not a version bump of EXL2 — this index does not track it yet, since no model here ships it, but a format this site tracks going quiet is itself worth knowing.

Speed tests use prompt_len=128, gen_len=128, single sequence. PPL measured on WikiText-2 test split. Your results may vary ±10% depending on driver, batch size, and context length.

Throughput

Prompt length: 128 tokens
Generation length: 128 tokens
Measurement (samples, mean/median, warm-up): not recorded
Reported as: generation tok/s, excluding prompt processing

Perplexity

FP16 baseline PPL: 6.14
Reported as: percentage of the baseline retained

VRAM

Measured with: not recorded
Reported as: peak usage during generation, including runtime overhead

Test hardware

NVIDIA cards: NVIDIA 550.x / CUDA 12.4. Apple cards: no CUDA/driver stack — Metal via each framework's own backend.

CPU / system RAM / OS per machine

not recorded

Runs recorded

not recorded

Last re-run

not recorded

Data source: Local inference runs on RTX 4090 / 3090 / M3 Max / M2 Ultra; llama.cpp b4000+, ExLlamaV2 0.2.x, vLLM 0.6.x, Ollama 0.3.x