EXL2 explained
What EXL2 is, which runtimes read it, what it costs in VRAM, and all 21 of the 81 models in this index that ship in it.
What EXL2 is
ExLlamaV2 format. Mixed-precision per-layer quantization — best accuracy-per-bit ratio. Fastest single-GPU inference available. 21 of the 81 models in this index ship in it, across 2 quant levels — 4.65bpw, 3.5bpw.
What reads it
ExLlamaV2 · TabbyAPI. Hardware: NVIDIA GPU (Ampere+ recommended). Best suited to: Max single-GPU performance. Where it is strong: Fastest GPU inference; Best accuracy per bit; Ultra-low 2bpw option. Where it is not: NVIDIA only; Steeper learning curve.
What it costs
Llama 3.1 8B Instruct at EXL2 4.65bpw is 4.65 bits per weight, which works out to 4.4 GB of weights plus 0.5 GB of KV cache at 4K context and a 0.5 GB activation buffer — 5.4 GB in total. The smallest card in this index that clears that comfortably is an RTX 5060 Ti 8G. Across the 15 models here with a published perplexity figure at this level, the median loss against FP16 is 2.2%.
EXL2 levels in this index
| Level | bits/weight | Models | Median loss |
|---|---|---|---|
| 4.65bpw | 4.65 | 15 | 2.2%median of 15 |
| 3.5bpw | 3.5 | 6 | 4.8%median of 6 |
Bits per weight is the model row's own figure, so a range means different models ship the same level at slightly different sizes. The median loss is computed only over the models with a published perplexity figure at that level; the sample size is printed beside it.
Models in this index that ship EXL2
21 of 81, largest first, each with its smallest EXL2 level and that level's published perplexity loss where one exists.
- Qwen2.5 72B Instruct72B · 3.5bpw · 4.8%
- Llama 3.1 70B Instruct70B · 3.5bpw · 5.2%
- Qwen3 32B Instruct32B · 3.5bpw · 4.5%
- Qwen2.5 32B Instruct32B · 3.5bpw · 4.8%
- Qwen2.5-Coder 32B Instruct32B · 3.5bpw · 4.5%
- DeepSeek-R1-Distill-Qwen-32B32B · 3.5bpw · 4.5%
- Mistral Small 24B Instruct24B · 4.65bpw · 2.2%
- Devstral Small 1.1 24B24B · 4.65bpw · 2.5%
- Magistral Small 1.2 24B24B · 4.65bpw · 2.5%
- Qwen3 14B Instruct14B · 4.65bpw · 1.8%
- Qwen2.5 14B Instruct14B · 4.65bpw · 2.1%
- DeepSeek-R1-Distill-Qwen-14B14B · 4.65bpw · 2.0%
- Qwen3 8B Instruct8B · 4.65bpw · 2.0%
- Llama 3.1 8B Instruct8B · 4.65bpw · 2.5%
- Nous Hermes 3 Llama 3.1 8B8B · 4.65bpw · 2.3%
- OpenChat 3.6 8B8B · 4.65bpw · 2.4%
- DeepSeek-R1-Distill-Llama-8B8B · 4.65bpw · 1.9%
- Qwen2.5 7B Instruct7B · 4.65bpw · 2.2%
- Qwen2.5-Coder 7B Instruct7B · 4.65bpw · 2.0%
- DeepSeek-R1-Distill-Qwen-7B7B · 4.65bpw · 2.2%
- Qwen3 4B Instruct4B · 4.65bpw · 2.1%
Common questions
What is EXL2?
ExLlamaV2 format. Mixed-precision per-layer quantization — best accuracy-per-bit ratio. Fastest single-GPU inference available. It is read by ExLlamaV2 · TabbyAPI, needs NVIDIA GPU (Ampere+ recommended), and 21 of the 81 models in this index ship in it.
How much VRAM does Llama 3.1 8B Instruct need in EXL2?
About 5.4 GB at EXL2 4.65bpw with a 4K context window — 4.4 GB of that is the weights themselves, so an 8 GB card clears it comfortably. Longer context adds KV cache on top; the calculator will size any combination.
Which EXL2 level should I download?
This index carries 4.65bpw (15 models), 3.5bpw (6 models). 4.65bpw is the one most models here ship and the usual starting point. 3.5bpw at 3.5 bpw is the smallest and 4.65bpw at 4.65 bpw the largest — the rule that matters is to take the highest level that still leaves headroom on your card, not the smallest one that loads.
EXL2 or something else?
Depends on what reads it — this one belongs where the job is max single-GPU performance. This site compares it directly against GGUF, AWQ, using only the models that ship both formats — the only place the comparison is measurable rather than editorial.