Local LLM FAQ

About 5.1 GB at Q4_K_M with a 4K context window — 4.2 GB of weights, 0.5 GB of KV cache and a 0.5 GB activation buffer, measured on Mistral 7B Instruct v0.3 (7B). Every 8 GB card in this index clears that with room left. The figure rises with context, not with how hard you use the model: the same file at 32K needs about 9.0 GB, because the KV cache grows to 4.0 GB.

At 4 bits and short context, yes — Mistral Small 24B Instruct at AWQ INT4 needs about 13.2 GB of 16 GB, leaving 2.8 GB. The level matters more than the parameter count: the same model at Q4_K_M is 15.9 GB, which is 99% of the card — it loads on a card with nothing else on it and has no margin for a longer window. A 16 GB card runs 52 of the 81 models here comfortably.

Yes. On a card that is also driving a desktop, subtract roughly 0.5–1.5 GB before comparing against any figure here — more with a browser open. That is why the calculator's "comfortable" verdict stops at 88% of the card rather than 100%: the last eighth is the margin that keeps a model from failing to allocate on a machine you are also using. The looser 105% rule is shown separately and always named.

Because a level like Q4_K_M is a recipe, not a fixed bit width — it quantizes different tensors to different precisions, and the mix depends on the model's shape. Across the 81 models here that ship Q4_K_M the effective rate runs from 4.1 to 4.9 bits per weight. This site stores each model's own measured rate rather than one table value, which is why two 8B models at the same level can differ by several hundred megabytes.

It depends on the model's attention shape, not on its parameter count. Mistral 7B Instruct v0.3 goes from 0.5 GB of KV cache at 4K to 4.0 GB at 32K — roughly linear, because every layer keeps a cache that grows with the window. 2026 hybrid-attention models break that: a model running only a fraction of its layers on full attention keeps a fixed-size recurrent state on the rest, so its cache barely moves. The calculator accounts for both.

At Q4_K_M the median published perplexity loss across the 79 models here that report one is 2.9% — 97.1% retained — with the range running 1.4% to 5.2%. Two caveats that matter more than the median: smaller models lose more than larger ones at the same level, and perplexity is a language-modelling measure, not a measure of whether the model still does your task. Models without a published figure are shown as a dash here rather than assumed to be lossless.

Q4 is the nominal bit width, K is llama.cpp's k-quant family, and M is the medium variant of it (there are S and L siblings). The K family does not quantize every tensor the same way — attention and feed-forward tensors get different treatment, and some stay at higher precision — which is why the effective rate lands above 4 bits. In this index Q4_K_M is the level every one of the 81 GGUF models ships, which is why it is the level the site quotes its headline quality figure against.

On the 16 models here that publish a figure at both levels, moving Q4_K_M → Q5_K_M recovers a median of 1.5 percentage points of perplexity loss for about 17% more file. Paired deliberately: the models reporting Q5 are not the same set as those reporting Q4, so two separate medians would compare the groups rather than the levels. In practice the decision is usually made for you by what fits — take the higher level when your card has the headroom, and do not drop below Q4 to squeeze in a bigger model without trying both.

This index cannot tell you. Every quality figure here is perplexity on WikiText-2, which measures next-token prediction on general prose — it is not a coding benchmark, and a site that cannot run task evaluations should not convert one into a claim about another. The widely reported pattern is that tasks with a single correct answer degrade more visibly than open-ended ones, but that is not something measured here, so it is not stated here as a number.

Usually yes at the same bit width, and the reason is mechanical: quantization-aware training runs the quantization in the forward pass during fine-tuning, so the weights adapt to it, where a community post-training quantization compresses a finished model and can only minimise the damage. What this site cannot yet tell you is which build you are looking at — the index records the format and level of each quant but not its publisher, so there is no "official" flag on any row here. That is a known gap, not a judgement that it does not matter.

GGUF unless you are running a server. GGUF runs on anything — CPU, NVIDIA, AMD, Apple — and 81 of the 81 models here ship it, against 53 for AWQ. AWQ's case is throughput under vLLM with batched requests on an NVIDIA card; for one person talking to one model, it buys you nothing GGUF does not already do. 53 models here ship both, which is the only set where the two can be compared on the same weights.

Not usefully. Both are lossy compressions of the original FP16 weights, so converting one to the other stacks a second round of loss on top of the first — you would be quantizing an already-quantized model. The path is to go back to the original weights and quantize those, which is what the people publishing each format already did. If a model has no build in the format you need, that is a real gap, not something a conversion fixes.

Because it is already 4-bit — the released checkpoint is itself the quantized one. The 2 models here shipping MXFP4 were released at 4 bits rather than converted down, so there is no FP16 original sitting behind them to recover. Re-quantizing to Q4_K_M costs quality and saves nothing: both land near the same size, and you have added a lossy step for it.

GGUF, through llama.cpp or Ollama with the Metal backend — 81 of the 81 models here. AWQ, EXL2 and GPTQ all require CUDA and will not run on Apple silicon at all, so three of the four formats this index tracks are unavailable to you before you start. The practical constraint on a Mac is not the format but how much of the unified memory the GPU is allowed to wire down, which is below the number on the box.

GGUF via llama.cpp's ROCm or Vulkan backend is the reliable answer. vLLM also ships official ROCm builds, so AWQ is not off the table the way it is on a Mac — but EXL2 and GPTQ are CUDA-only. The thing that actually decides whether an AMD setup works is not the format but whether your kernel and card have a working ROCm build, which is a setup question rather than a hardware ceiling.

Whichever one has the most memory in your budget — capacity decides what you can run at all, and nothing else does. Of the 61 cards in this index the largest consumer option is the RTX 5090 at 32 GB and 1,792 GB/s, which runs 66 of the 81 models here comfortably against 35 for an 8 GB card. Bandwidth is the second question, not the first: it sets how fast a model that already fits will generate, and two cards with the same memory can differ by more than 2.5× on it.

For a large part of what people actually run, yes — 35 of the 81 models in this index fit an 8 GB card comfortably at 4K context, including 7B-class models at Q4_K_M (5.1 GB). What 8 GB does not give you is headroom: a long context window, a second process, or a model above about 13B. The step to 16 GB takes you from 35 models to 52.

Mostly, but not all of it. The GPU addresses the same pool as the CPU, so a 64 GB Mac can hold models a 24 GB discrete card cannot — that part is real. What is not real is treating the whole number as available: macOS reserves part of the pool for the system and caps how much a single process may wire down. Budget meaningfully below the figure on the box, and remember bandwidth differs by tier far more than capacity suggests.

One 32 GB card, for a single model. Splitting a model across two cards means every token crosses the bus between them, and the layers on the second card wait on the first — you get the capacity but not the throughput, and the setup is materially harder. Two cards earn their place when you want to run two things at once, or when a 32 GB card is not in the budget. A 32 GB card runs 66 of the 81 models here comfortably against 52 for a single 16 GB card.

Not on a consumer graphics card, no. Llama 3.1 70B Instruct at Q4_K_M needs about 46.1 GB, and no consumer GPU in this index has that much memory — the cards that do are 13 datacentre, Apple and CPU entries. Its smallest build here, EXL2 3.5bpw, still needs 33.6 GB. The realistic routes are a large-memory Mac, system RAM with CPU inference and the speed that implies, or two cards.

Three sources, kept apart. Model architecture (layer count, KV heads, head dimension) comes from each model's own published config and drives the VRAM arithmetic. Perplexity figures are the ones the quantizers published; models without one show a dash rather than a guess. Speed figures are runs recorded on Meta Llama 3.1 8B Instruct against WikiText-2 on a named stack — and only on hardware that stack actually ran on. Everything else on the site, including every VRAM number, is calculated rather than observed, and labelled as such.

Measured means someone ran it and recorded the number on named hardware with a named runtime version. Estimated means it was computed from the model's architecture and quant level using the same formula the calculator uses. The site labels which is which on every surface that shows a figure, and a hardware page with no runs on it says so rather than letting its estimates read as measurements. Removing a row is preferred to adjusting one — three benchmark rows were deleted after arithmetic showed the card they named could not physically reach the speed claimed.

The data carries its own date — it currently reads 2026-09-11 — and every change that alters what the site says is written into a changelog entry on the same day. The rhythm is a data refresh most weeks and a batch of new models every couple of weeks, weighted toward models a constrained setup can actually run rather than toward whatever launched loudest.

Check three things first: the context length (the default here is 4K and the KV cache grows with it), whether your card is also driving a display, and whether your build is the same quant level rather than the same nominal bit width. If it still disagrees, that is worth reporting — a figure that does not survive contact with real hardware is a figure this site would rather fix than defend, and three benchmark rows have already been removed for exactly that reason.