IntermediateEdge / Local 10 min read

ExLlamaV2 on RTX 4090: Full Setup Guide

Install ExLlamaV2, load an EXL2 quant, and serve an OpenAI-compatible API in under 10 minutes.

Verified stack

Python 3.10+ · CUDA 12.x · ExLlamaV2 · EXL2 model dir · RTX 4090

Last verified 2026-07-22

ExLlamaV2EXL2RTX 4090API

Install

ExLlamaV2 requires NVIDIA CUDA. Python 3.10+ recommended.

bash

pip install exllamav2
# Or clone for latest:
git clone https://github.com/turboderp/exllamav2
cd exllamav2 && pip install -r requirements.txt

Run inference

Point at a downloaded EXL2 model directory. Adjust -c for context length.

bash

python examples/chat.py \
  -m ./models/Llama-3.1-8B-exl2-4.65bpw \
  -c 4096 -gs 16

Related guides

IntermediateEdge / Local9 min read

DeepSeek-R1 Distill 14B: EXL2 vs GGUF

Head-to-head on RTX 4090 — when to pick turboderp EXL2 over bartowski GGUF.

IntermediateEdge / Local8 min read

Qwen2.5-Coder 32B on a Single RTX 4090

The best open coding model that fits in 24GB — quant selection and tuning tips.

BeginnerEdge / Local7 min read

What Can You Run on RTX 4060 Ti 16G?

A practical guide to picking the right model and quant level for NVIDIA's best budget 16GB card.

Deployment guides are educational. Each model is subject to its own license — read the official Hugging Face model card before downloading or deploying.