What you need first, and what this is not
An NVIDIA card and a working EXL2 model directory — TabbyAPI is a wrapper, so everything the ExLlamaV2 guide requires applies here unchanged. TabbyAPI wraps ExLlamaV2 in an OpenAI-compatible server, which is what makes EXL2 usable from editor plugins and desktop clients that only speak that protocol. Read the project’s own framing before you build on it: its README states plainly that it is a hobby project for a small number of users and **is not meant to run on production servers**. That is the maintainers’ assessment, not this site’s — take it at face value and use vLLM where you need a service.
git clone https://github.com/theroyallab/tabbyAPI
cd tabbyAPIInstall and configure
Python 3.10 through 3.14 is supported. The start scripts set up a virtual environment and pull the right ExLlamaV2 wheel for your platform, which is the part worth letting them do — matching wheels by hand is where most failed installs come from. Configuration lives in `config.yml`, copied from the sample.
cp config_sample.yml config.yml
# Linux / macOS
./start.sh
# Windows
start.batPoint it at a model
EXL2 models go in `models/` as directories — one folder per model, not one file. Set the model name and the context length in `config.yml`; the context length is the setting that decides whether the thing loads, because the KV cache scales with it.
model:
model_dir: models
model_name: Llama-3.1-8B-exl2-4.65bpw
max_seq_len: 4096
network:
host: 127.0.0.1
port: 5000Check it actually worked
Ask the OpenAI-compatible endpoints, the same way you would check any other server of this shape. If the models endpoint answers, weights are loaded and the cache is allocated.
curl http://127.0.0.1:5000/v1/models
curl http://127.0.0.1:5000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"messages":[{"role":"user","content":"hi"}]}'
nvidia-smi --query-gpu=memory.used --format=csvWhat the numbers should look like
The backend is ExLlamaV2, so the throughput is ExLlamaV2’s: this index measured Llama 3.1 8B at EXL2 4.65bpw at 235 tok/s on an RTX 4090 and 175 tok/s on an RTX 3090. Memory is 5.4 GB at 4K context and 9.3 GB at 32K for that model — and `max_seq_len` in the config is what picks which of those you pay.
When it does not work
The server starts but no model loads: `model_name` must match the directory name under `models/` exactly, and it is a directory rather than a file. Out of memory on load: lower `max_seq_len` first — the cache scales with it and the weights do not. A wheel or import error: let `start.sh` build the environment rather than installing ExLlamaV2 yourself into an existing one, since it resolves the wheel against your Python and CUDA. And if you are reaching for authentication, rate limiting or multi-tenancy, that is the signal the README already gave — this is not the project for it.
Common questions
Is TabbyAPI suitable for production?
Its own README says no: "a hobby project made for a small amount of users… not meant to run on production servers." That is the maintainers’ statement and it is worth respecting. Use it to make EXL2 reachable from editor plugins and desktop clients on your own machine; use vLLM when the thing you are building is a service.
Why will my model not load in TabbyAPI?
The two usual causes are the name and the window. `model_name` has to match the directory under `models/` exactly — EXL2 models are folders, not single files — and `max_seq_len` decides how much KV cache is allocated, so a window larger than your card can hold fails at load rather than at request time.
What throughput should I see through TabbyAPI?
Whatever ExLlamaV2 gives, since it is the backend. On this index’s measured runs, Llama 3.1 8B at EXL2 4.65bpw reached 235 tok/s on an RTX 4090 and 175 tok/s on an RTX 3090. The HTTP layer is not where the time goes; the card’s memory bandwidth is.
What this guide uses
- Hardware
- RTX 409024GBRTX 3090
- Models
- Llama 3.1 8B Instruct8B
- Format
- EXL2
- Picks for it
- Best local LLM for 24GB
Next steps
See everything that fits 🟢 RTX 4090Reverse lookup — 24GB at 8192 context, ranked by qualityModels covered in this guide
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
Multi-Model API Server on RTX 4090 with vLLM
Serve multiple AWQ-quantized models with vLLM's continuous batching for production-grade throughput.
Nginx Reverse Proxy for Local LLM APIs
Put Ollama or llama.cpp behind Nginx with TLS, rate limiting, and a stable /v1 endpoint for your apps.
Run Llama 3.1 8B on a €20/month VPS
A complete guide to running a private LLM API on a budget Linux VPS using llama.cpp server mode.