Phi-3.5 Mini Instruct

Running Phi-3.5 Mini Instruct locally

At Q4_K_M and 4K of context, Phi-3.5 Mini Instruct needs about 4.1 GB — 2.2 GB of weights, 1.50 GB of KV cache and a 0.4 GB activation buffer. The smallest card in this index that clears that comfortably is the RTX 5060 Ti 8G at 8 GB, and 63 of the 63 cards here do. These are calculated figures, not measurements: the estimate stops counting a card as comfortable at 88% of its VRAM, which is roughly the room a desktop session needs.

What longer context costs

Going from 4K to 32K adds about 10.50 GB, taking the total to 15.6 GB. Weights do not move with context — only the KV cache does, and it grows linearly, so this is the number to watch when planning for long documents. The model's native window is 128K; holding all of it at Q4_K_M would need about 55 GB.

Which build to download

This index tracks 2 formats for it — GGUF, AWQ — across 3 levels. Q8_0 carries the lowest published perplexity loss at 0.2%. The fastest level measured here is AWQ INT4 at 385 tok/s on an RTX 4090, batch 1. GGUF runs on llama.cpp and Ollama across NVIDIA, AMD and Apple silicon; AWQ and GPTQ target vLLM on CUDA and ROCm; EXL2 is ExLlamaV2 and CUDA only.

Common questions

How much VRAM does Phi-3.5 Mini Instruct need?

About 4.1 GB at Q4_K_M with 4K of context and batch 1, rising to roughly 15.6 GB at 32K. That figure is weights plus KV cache plus a 10% activation buffer, calculated from the model's architecture rather than measured on a card.

Will Phi-3.5 Mini Instruct run on a 8GB GPU?

Yes — at Q4_K_M and 4K context it needs about 4.1 GB, which leaves 3.9 GB spare on a RTX 5060 Ti 8G. That is the smallest card in this index that clears it comfortably; 63 of 63 do.

Which quantization of Phi-3.5 Mini Instruct should I use?

Q8_0 has the lowest published quality loss (0.2%), and Q4_K_M is the level most people run. All 3 levels in the index are Q4_K_M, Q8_0, AWQ INT4.