Install TabbyAPI
TabbyAPI is the most popular ExLlamaV2 server wrapper with a built-in UI.
bash
git clone https://github.com/theroyallab/tabbyAPI
cd tabbyAPI
pip install -r requirements.txtConfigure and run
Place EXL2 models in the models/ directory and start the API server.
bash
python main.py --port 5000
# API: http://localhost:5000/v1/chat/completionsRelated guides
IntermediateServer / VPS12 min read
Multi-Model API Server on RTX 4090 with vLLM
Serve multiple AWQ-quantized models with vLLM's continuous batching for production-grade throughput.
IntermediateServer / VPS11 min read
Nginx Reverse Proxy for Local LLM APIs
Put Ollama or llama.cpp behind Nginx with TLS, rate limiting, and a stable /v1 endpoint for your apps.
BeginnerServer / VPS8 min read
Run Llama 3.1 8B on a €20/month VPS
A complete guide to running a private LLM API on a budget Linux VPS using llama.cpp server mode.
Deployment guides are educational. Each model is subject to its own license — read the official Hugging Face model card before downloading or deploying.