Mistral Large 3 675B Instruct

Running Mistral Large 3 675B Instruct locally

At Q4_K_M and 4K of context, Mistral Large 3 675B Instruct needs about 428.7 GB — 388.7 GB of weights, 1.00 GB of KV cache and a 39.0 GB activation buffer. No card in this index clears that comfortably, so it is a multi-GPU or CPU-offload proposition. These are calculated figures, not measurements: the estimate stops counting a card as comfortable at 88% of its VRAM, which is roughly the room a desktop session needs.

What longer context costs

Going from 4K to 32K adds about 7.00 GB, taking the total to 436.4 GB. Weights do not move with context — only the KV cache does, and it grows linearly, so this is the number to watch when planning for long documents. The model's native window is 256K; holding all of it at Q4_K_M would need about 498 GB.

Which build to download

This index only tracks Mistral Large 3 675B Instruct in GGUF, across 2 levels (Q4_K_M, Q3_K_M). Q4_K_M carries the lowest published perplexity loss at 2.1%. The fastest level measured here is Q3_K_M at 5 tok/s on an RTX 4090, batch 1. GGUF runs on llama.cpp and Ollama across NVIDIA, AMD and Apple silicon; AWQ and GPTQ target vLLM on CUDA and ROCm; EXL2 is ExLlamaV2 and CUDA only.

Common questions

How much VRAM does Mistral Large 3 675B Instruct need?

About 428.7 GB at Q4_K_M with 4K of context and batch 1, rising to roughly 436.4 GB at 32K. That figure is weights plus KV cache plus a 10% activation buffer, calculated from the model's architecture rather than measured on a card.

Can Mistral Large 3 675B Instruct run on a single GPU?

Not comfortably on any single card in this index at Q4_K_M. It needs about 428.7 GB, which means splitting across GPUs or offloading layers to system RAM.

Which quantization of Mistral Large 3 675B Instruct should I use?

Q4_K_M has the lowest published quality loss (2.1%), and Q4_K_M is the level most people run. All 2 levels in the index are Q4_K_M, Q3_K_M.