What you need first
An NVIDIA card — ExLlamaV2 is CUDA-only, with no ROCm or Metal path, so this is the one runtime where the hardware question is settled before anything else. Ampere or newer is recommended. Python 3.10 or later, and a CUDA 12.x toolchain if you build from source. The repository now lives under `turboderp-org`; the old `turboderp/exllamav2` URL still redirects, which is why half the tutorials online still use it.
pip install exllamav2
# Or from source, which is what you want if you need the latest kernels:
git clone https://github.com/turboderp-org/exllamav2
cd exllamav2
pip install -r requirements.txt
pip install .Get an EXL2 model
EXL2 is a directory, not a single file — this trips people who arrive from GGUF expecting one `.gguf` to download. Quantization is per-layer and the bit-rate is a continuous dial rather than a menu, which is where the format’s accuracy-per-bit advantage comes from. This index carries 21 models with an EXL2 build, mostly at 4.65bpw.
pip install huggingface_hub
huggingface-cli download turboderp/Llama-3.1-8B-Instruct-exl2 \
--revision 4.65bpw \
--local-dir ./models/Llama-3.1-8B-exl2-4.65bpwRun inference
The bundled chat example is the quickest way to confirm the install. `-gs auto` lets it work out the split across your cards, which is the current documented form — older tutorials pass explicit per-GPU gigabyte numbers like `-gs 16`, and those are only worth writing by hand when you deliberately want to reserve memory on one card.
python examples/chat.py \
-m ./models/Llama-3.1-8B-exl2-4.65bpw \
-mode llama \
-gs autoCheck it actually worked
ExLlamaV2 has no CPU fallback, so unlike llama.cpp it fails loudly rather than running slowly — which is genuinely convenient. The check that matters is memory: watch the card while the model loads, and compare what it takes against what it should take.
nvidia-smi --query-gpu=memory.used --format=csv -l 1
# Llama 3.1 8B at 4.65bpw should settle around 5.4 GB at 4K context.
# Substantially more usually means the context length is larger than you think.What the numbers should look like
Llama 3.1 8B at EXL2 4.65bpw is 4.4 GB of weights and 5.4 GB in total at 4K context, rising to 9.3 GB at 32K — the KV cache nearly matches the weights by then. This index has measured runs for exactly this configuration: **235 tok/s on an RTX 4090** and **175 tok/s on an RTX 3090**. That 4090 figure sits essentially at the bandwidth ceiling for this file size, which is what ExLlamaV2 is known for and why it beats the same model’s GGUF build on the same card.
When it does not work
An import error about a compiled extension: the wheel does not match your Python or CUDA version — installing from source with `pip install .` builds against what you actually have, at the cost of a long compile. Out of memory at a context length that used to load: the KV cache scales with the window, and at 32K this model’s cache is almost as large as its weights, so the fix is the context rather than the quant. A model directory that will not load at all: check it is an EXL2 conversion rather than the original weights — the directory layouts look similar and only one of them has the quantized tensors. And there is no CPU offload here by design, so "it fits with a little help from system RAM" is not an option the way it is in llama.cpp.
Common questions
Is ExLlamaV2 faster than llama.cpp?
On a single NVIDIA card, on this index’s own measurements, yes: Llama 3.1 8B on an RTX 4090 measured 235 tok/s under ExLlamaV2 at EXL2 4.65bpw against 148 tok/s under llama.cpp at GGUF Q4_K_M. The trade is reach — ExLlamaV2 is CUDA-only, while llama.cpp runs on CPU, AMD and Apple silicon as well.
Why is an EXL2 model a folder instead of one file?
Because the quantization is per-layer with a continuous bit-rate, so the format keeps the model’s structure rather than packing everything into a single container the way GGUF does. Download a specific bit-rate with `--revision` — the same repository holds several, and grabbing the default gets you whichever the publisher made the main branch.
Can I run EXL2 on an AMD card or a Mac?
No. ExLlamaV2 is CUDA-only — there is no ROCm build and no Metal path, so the format is unavailable on AMD and Apple silicon regardless of how much memory you have. GGUF is the format that runs everywhere, and 81 of the 81 models in this index ship it.
What this guide uses
- Hardware
- RTX 409024GBRTX 3090
- Models
- Llama 3.1 8B Instruct8B
- Format
- EXL2
- Picks for it
- Best local LLM for 24GB
Next steps
See everything that fits 🟢 RTX 4090Reverse lookup — 24GB at 8192 context, ranked by qualityModels covered in this guide
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
DeepSeek-R1 Distill 14B: EXL2 vs GGUF
Head-to-head on RTX 4090 — when to pick turboderp EXL2 over bartowski GGUF.
Qwen2.5-Coder 32B on a Single RTX 4090
The best open coding model that fits in 24GB — quant selection and tuning tips.
What Can You Run on RTX 4060 Ti 16G?
A practical guide to picking the right model and quant level for NVIDIA's best budget 16GB card.