AdvancedServer / VPS 10 min read

vLLM + AWQ in Production: Tuning Guide

gpu-memory-utilization, max-model-len, and batching knobs for stable API serving.

Verified stack

vLLM 0.6+ · AWQ · gpu-memory-utilization 0.75–0.85 · max-model-len tuned

Last verified 2026-07-22

vLLMAWQproductionAPI

Memory tuning

Start at 0.85 gpu-memory-utilization. Lower to 0.75 if you see OOM on long contexts.

bash

python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-7B-Instruct-AWQ \
  --quantization awq \
  --gpu-memory-utilization 0.85 \
  --max-model-len 32768 \
  --max-num-seqs 32

Related guides

IntermediateServer / VPS12 min read

Multi-Model API Server on RTX 4090 with vLLM

Serve multiple AWQ-quantized models with vLLM's continuous batching for production-grade throughput.

BeginnerServer / VPS8 min read

Run Llama 3.1 8B on a €20/month VPS

A complete guide to running a private LLM API on a budget Linux VPS using llama.cpp server mode.

IntermediateServer / VPS8 min read

TabbyAPI: ExLlamaV2 with a Web UI

Wrap ExLlamaV2 in TabbyAPI for a polished OpenAI-compatible server with streaming and model hot-swap.

Deployment guides are educational. Each model is subject to its own license — read the official Hugging Face model card before downloading or deploying.