Benchmarks & Insights

Name: quantized.uk hardware benchmark matrix
Creator: quantized.uk
License: https://creativecommons.org/licenses/by/4.0/

Hard numbers, no hype — measured on real hardware

Every row below was produced on a machine described on this page, not collected from elsewhere. That also means the coverage is narrow: 16 runs across 4 cards, out of 78 in the GPU index. Where a figure on this site is not measured, it says estimated — this page is the only place the measured ones come from.

The benchmark data on this page is licensed CC BY 4.0 — reuse it freely with credit and a link to quantized.uk. The whole index is also published as JSON (models and GPUs).

The same figures, as text

Charts are drawn as SVG paths, so the numbers in them are not readable text. These tables carry the same data.

What is and is not covered

Measured here

Estimated only

The other 78 cards in the index. Their numbers come from the same formula the VRAM calculator uses, scaled by published memory bandwidth — not a second, hidden measurement.

Not covered

Multi-GPU throughput beyond the dual-RTX-3090 guide
Batch sizes above 1
Prompt processing (prefill) speed, as distinct from generation
Quality on anything other than perplexity

Hardware × Format Matrix

Multi-model speed & VRAM on real hardware — same quant levels where comparable

Hardware × Format Matrix — Multi-model speed & VRAM on real hardware — same quant levels where comparable
Model	Hardware	Framework	Quant	Speed (tok/s)	VRAM Used	Notes
Llama 3.1 8B	RTX 4090 24G	ExLlamaV2	EXL2 4.65bpw	235	5.4 GB	Peak consumer performance
Llama 3.1 8B	RTX 4090 24G	vLLM	AWQ INT4	218	4.9 GB	Best for batch API
Llama 3.1 8B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	148	5.7 GB	Easiest setup
Qwen2.5 7B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	155	5.4 GB	Strong coding; similar VRAM to 8B
DeepSeek-R1 14B	RTX 4090 24G	ExLlamaV2	EXL2 4.65bpw	128	9.8 GB	Reasoning distill; hot in 2026
Qwen2.5 32B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	44	22 GB	Tight fit at 4K ctx; use Q3 for headroom
Qwen3 8B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	142	5.8 GB	Qwen3 thinking mode; ~2026 flagship 8B
Qwen3 14B	RTX 4090 24G	ExLlamaV2	EXL2 4.65bpw	118	10 GB	Strong reasoning; 16GB+ sweet spot
Qwen3 32B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	42	22.5 GB	Dense 32B successor to Qwen2.5-32B
Qwen3 30B-A3B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	95	19 GB	MoE 3B active — fast on 16GB cards
R1-Distill-Llama-8B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	145	5.6 GB	R1 reasoning on 8B footprint
Phi-4 14B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	88	9.1 GB	Strong dense 14B; 12GB with short ctx
Qwen3-Coder 30B-A3B	RTX 4090 24G	llama.cpp	GGUF Q4_K_M	92	19.2 GB	Agentic coding MoE; 256K native ctx
Llama 3.1 8B	RTX 3090 24G	ExLlamaV2	EXL2 4.65bpw	175	5.4 GB	Older but capable
Llama 3.1 8B	M3 Max 48G	Ollama	GGUF Q4_K_M	68	5.7 GB	Unified memory advantage
Llama 3.1 8B	M2 Ultra 192G	llama.cpp	GGUF Q4_K_M	90	5.7 GB	Can run 70B models solo

What counts as a measured run here

The rules every row in the table below passes. The last check is enforced by the build: a row that fails it cannot ship.

One sequence at a time (batch 1): a 128-token prompt, 128 generated tokens, a 4096-token context.
The runtime and its version are recorded with the run — they are listed under Test Methodology.
VRAM is the peak used during generation, runtime overhead included.
Every dense-model run is checked against its card's bandwidth ceiling. Generating a token reads every weight once, so bandwidth divided by the weight size is a hard limit; a run more than 15% over it is rejected. Three RTX 4060 Ti rows were removed for this.
Mixture-of-experts runs read only their active experts per token, so they are not held to the dense ceiling.
Anything elsewhere on the site that does not come from this page is labelled estimated.

What these runs do not include: the raw benchmark output and the full launch flags were not kept. Any new run, including one you send, has to come with its raw output.

Recorded VRAM next to the calculator

For each run, the calculator's estimate for the same model and level at the same 4096-token context. This is how far the site's own estimates can be trusted.

Across the 15 runs the index can size, the median gap is 0.2 GB. The largest is Phi-4 14B: the calculator says 10.2 GB, the run recorded 9.1 GB.

Recorded VRAM next to the calculator
Model	Card	Level	Recorded VRAM	Calculator	Difference	tok/s	Ceiling (share)
Llama 3.1 8B	RTX 4090	EXL2 4.65bpw · ExLlamaV2	5.4 GB	5.4 GB	0.0	235	228 (103%)
Llama 3.1 8B	RTX 4090	AWQ INT4 · vLLM	4.9 GB	4.8 GB	−0.2	218	265 (82%)
Llama 3.1 8B	RTX 4090	GGUF Q4_K_M · llama.cpp	5.7 GB	5.6 GB	−0.1	148	218 (68%)
Qwen2.5 7B	RTX 4090	GGUF Q4_K_M · llama.cpp	5.4 GB	5.1 GB	−0.3	155	230 (67%)
DeepSeek-R1 14B	RTX 4090	EXL2 4.65bpw · ExLlamaV2	9.8 GB	not in the index under this name	—	128	—
Qwen2.5 32B	RTX 4090	GGUF Q4_K_M · llama.cpp	22.0 GB	21.7 GB	−0.3	44	54 (81%)
Qwen3 8B	RTX 4090	GGUF Q4_K_M · llama.cpp	5.8 GB	5.8 GB	0.0	142	214 (66%)
Qwen3 14B	RTX 4090	EXL2 4.65bpw · ExLlamaV2	10.0 GB	9.7 GB	−0.3	118	124 (95%)
Qwen3 32B	RTX 4090	GGUF Q4_K_M · llama.cpp	22.5 GB	21.9 GB	−0.6	42	53 (79%)
Qwen3 30B-A3B	RTX 4090	GGUF Q4_K_M · llama.cpp	19.0 GB	19.7 GB	+0.7	95	—
R1-Distill-Llama-8B	RTX 4090	GGUF Q4_K_M · llama.cpp	5.6 GB	5.6 GB	0.0	145	218 (67%)
Phi-4 14B	RTX 4090	GGUF Q4_K_M · llama.cpp	9.1 GB	10.2 GB	+1.1	88	119 (74%)
Qwen3-Coder 30B-A3B	RTX 4090	GGUF Q4_K_M · llama.cpp	19.2 GB	19.8 GB	+0.6	92	—
Llama 3.1 8B	RTX 3090	EXL2 4.65bpw · ExLlamaV2	5.4 GB	5.4 GB	0.0	175	211 (83%)
Llama 3.1 8B	Mac M3 Max 48G	GGUF Q4_K_M · Ollama	5.7 GB	5.6 GB	−0.1	68	87 (78%)
Llama 3.1 8B	Mac M2 Ultra 192G	GGUF Q4_K_M · llama.cpp	5.7 GB	5.6 GB	−0.1	90	173 (52%)

The ceiling is the figure the GPU pages quote. It divides decimal GB/s by binary GiB of weights, so it runs about 7% low — which is why a run can sit just above 100%. Mixture-of-experts rows show no ceiling.

Send a run from your own card

This page has runs on 4 cards. If yours is not one of them — 16 to 24 GB cards especially — one run can fill the gap. It can only be added if it can be checked, so run exactly this:

llama-bench -m <model-file>.gguf -p 512 -n 128 -ngl 99

The full output, unedited — it carries the build line and the pp512 / tg128 table.
The exact GGUF file name and the repository you downloaded it from.
Your card (or Mac chip and memory), driver version and operating system.
Nothing else using the GPU while it runs.

It is checked against your card's bandwidth ceiling like every row above. llama-bench's tg128 generates from an empty context, which is close to but not the same as the method above, so a reader run is marked as one. Only the card is published — never your name or address.

The email it opens, before you add anything:

Subject: quantized.uk benchmark run

Card / chip and memory:
Driver and OS:
GGUF file:
Downloaded from:

llama-bench output (unedited):

Open this email

Reference Model

Meta Llama 3.1 8B Instruct

PPL Dataset

WikiText-2

Context

4096 tokens

Batch Size

Framework Versions

llamacpp: b4217 (CUDA backend)exllama: ExLlamaV2 0.2.1vllm: v0.6.3ollama: 0.3.14

Current releases of those runtimes (checked 2026-11-07)

llamacpp: b12180vllm: v0.33.0ollama: v0.40.15exllama: v0.3.2 (stalled)

The figures on this page were measured on the stack in the left column. Those releases are now well behind current — speed numbers in particular move with the runtime, so treat them as a ranking between formats rather than as what you will see today. One thing this comparison cannot show: ExLlamaV2 has had no release since 2025-07, and the maintainer's active project has moved to a separate repository, ExLlamaV3 (v1.6.6 as of the date above, and shipping fast). Its own README describes a genuinely new quantization format, EXL3, not a version bump of EXL2 — this index does not track it yet, since no model here ships it, but a format this site tracks going quiet is itself worth knowing.

Speed tests use prompt_len=128, gen_len=128, single sequence. PPL measured on WikiText-2 test split. Your results may vary ±10% depending on driver, batch size, and context length.

Throughput

Prompt length: 128 tokens
Generation length: 128 tokens
Measurement (samples, mean/median, warm-up): not recorded
Reported as: generation tok/s, excluding prompt processing

Perplexity

FP16 baseline PPL: 6.14
Reported as: percentage of the baseline retained

VRAM

Measured with: not recorded
Reported as: peak usage during generation, including runtime overhead

Test hardware

NVIDIA cards: NVIDIA 550.x / CUDA 12.4. Apple cards: no CUDA/driver stack — Metal via each framework's own backend.

CPU / system RAM / OS per machine

not recorded

Runs recorded

not recorded

Last re-run

not recorded

Data source: Local inference runs on RTX 4090 / 3090 / M3 Max / M2 Ultra; llama.cpp b4217, ExLlamaV2 0.2.1, vLLM v0.6.3, Ollama 0.3.14