Quant Hub

Meta Llama 3.1

Meta's flagship 8B model with 128K context. Best-in-class for local deployment.

Formats

Meta Llama 3.1

Meta's frontier 70B model. Requires 40GB+ VRAM; dual 3090 or M2 Ultra.

Formats

Meta Llama 3.2

Tiny but capable. Runs on 4GB VRAM or 8GB RAM, even on phones via llama.cpp.

Formats

Alibaba Qwen2.5

Alibaba's highly optimized 7B. Punches well above its weight, especially in coding.

Formats

Alibaba Qwen2.5

The sweet spot between performance and resource usage. 16GB VRAM with Q4.

Formats

Alibaba Qwen2.5

Near-GPT-4 reasoning on a 24GB VRAM card (Q4_K_S). Groundbreaking value.

Formats

DeepSeek

MoE architecture coding model. Active params ~2.4B, total ~16B. Exceptional code quality. Its attention is MLA: a GGUF converted by current llama.cpp (tensors attn_k_b / attn_v_b) caches about 1 GB at 32K, which is what the estimates here assume. An older conversion with a single attn_kv_b tensor caches full keys and values instead — about 8.4 GB at 32K.

Formats

Microsoft Phi

Microsoft's tiny powerhouse. Best 4B model for on-device deployment.

Formats

Mistral AI

Mistral + NVIDIA collaboration. 128K context, excellent multilingual support.

Formats

Alibaba Qwen2.5

Flagship Qwen2.5. Requires dual 4090 or A100 80G. Exceptional reasoning at scale.

Formats

DeepSeek

R1 reasoning distilled into 14B. Huge community interest; excellent chain-of-thought.

Formats

Meta Llama 3.3

Latest Meta 70B with improved multilingual. Drop-in upgrade from Llama 3.1 70B.

Formats

Mistral AI

Mistral's efficient 24B. Strong multilingual; fits on 24GB with Q4.

Formats

Alibaba Qwen2.5

Top-tier open coding model. HumanEval competitive with GPT-4o on 32B scale.

Formats

Alibaba Qwen2.5

Best 7B coding model. Ideal for local dev assistants on 8–16GB VRAM.

Formats

Alibaba Qwen2.5

Tiny Qwen2.5 for edge devices. Runs on 4GB VRAM or Raspberry Pi class hardware.

Formats

Meta Llama 3.2

Ultra-light Llama for mobile and embedded. Sub-2GB VRAM with Q4.

Formats

DeepSeek

R1 reasoning in Llama 70B architecture. Top open reasoning model for dual-GPU setups.

Formats

Mistral AI

Mistral's dedicated code model. 80+ language support, Fill-in-the-Middle capable.

Formats

Meta Llama 3.2

Multimodal Llama with image understanding. Vision encoder adds ~2GB VRAM overhead.

Formats

NousResearch

Fine-tuned Llama 3.1 8B with improved roleplay and instruction following.

Formats

IBM Granite

IBM's enterprise-grade 8B. Strong RAG and tool-use; permissive Apache 2.0 license.

Formats

Alibaba Qwen2.5

Smallest Qwen2.5. Ideal for Raspberry Pi, phones, and ultra-low-latency demos.

Formats

Alibaba Qwen2.5

Tiny Qwen with 128K context. Surprisingly capable for summarisation and chat.

Formats

Microsoft Phi

Microsoft's mid-size Phi-3. Excellent quality-per-GB on 16GB cards.

Formats

Microsoft Phi

Latest Phi mini with improved math and code. Strong 4B-class performer.

Formats

DeepSeek

MoE general model (~2.4B active). Long context and strong multilingual chat. Its attention is MLA: a GGUF converted by current llama.cpp (tensors attn_k_b / attn_v_b) caches about 1 GB at 32K, which is what the estimates here assume. An older conversion with a single attn_kv_b tensor caches full keys and values instead — about 8.4 GB at 32K.

Formats

DeepSeek

R1 reasoning in a 7B footprint. Best value for 8–12GB VRAM CoT experiments.

Formats

DeepSeek

R1 distilled to 32B. Near-frontier reasoning on a single 24GB card (Q3/Q4).

Formats

Meta Llama 3.2

Flagship multimodal Llama. Requires dual 4090 or A100; vision adds ~3GB overhead.

Formats

TII UAE

Technology Innovation Institute's latest Falcon. Good multilingual and code mix.

Formats

Alibaba Qwen3

Latest Qwen3 dense 8B with thinking mode. Strong upgrade from Qwen2.5 7B for local deploy.

Formats

Alibaba Qwen3

Qwen3 14B — best balance of reasoning and VRAM in the 2026 Qwen lineup.

Formats

Google Gemma 3

Google Gemma 3 multimodal 4B. 128K context; strong vision + text on 8GB cards.

Formats

Google Gemma 3

Mid-size Gemma 3 with vision. Fits 16GB at Q4; excellent multilingual chat.

Formats

Meta Llama 4

Meta Llama 4 Scout MoE (17B active / 109B total). Multimodal; needs ~68GB VRAM at Q4_K_M.

Formats

Meta Llama 4

Llama 4 Maverick flagship MoE (17B active / 400B total). Multi-GPU or H100 cluster territory.

Formats

Meta Llama 3.1

Meta frontier dense 405B. Q4 needs ~230GB+ VRAM; dual H100 80G or 8× consumer GPU.

Formats

Alibaba Qwen3

Qwen3 dense 32B — successor to Qwen2.5-32B with stronger reasoning and thinking mode.

Formats

Alibaba Qwen3

Qwen3 MoE with only 3B active params. Q4 ~19GB file; outperforms QwQ-32B on 16GB cards.

Formats

Alibaba Qwen3

Qwen3 flagship MoE (22B active / 235B total). Q4_K_M ~142GB; rivals DeepSeek-R1 class models.

Formats

DeepSeek

DeepSeek-V3 frontier MoE (~37B active / 671B total). MLA + FP8; multi-node GPU cluster required at Q4. MLA cache: a GGUF converted by current llama.cpp (attn_k_b / attn_v_b) holds about 2 GB at 32K, which the estimates here assume; an older conversion with a single attn_kv_b tensor caches full keys and values — about 150 GB at 32K.

Formats

DeepSeek

DeepSeek-R1 reasoning model built on V3 MoE. Chain-of-thought at frontier scale — use distill variants for local GPUs. MLA cache: a GGUF converted by current llama.cpp (attn_k_b / attn_v_b) holds about 2 GB at 32K, which the estimates here assume; an older conversion with a single attn_kv_b tensor caches full keys and values — about 150 GB at 32K.

Formats

Alibaba Qwen3

Smallest Qwen3 dense with thinking mode. Q4 ~3.2GB — ideal for 8GB GPUs and edge devices.

Formats

Alibaba Qwen3

Agentic coding MoE with 3.3B active params and 256K native context. Top open coder for 16–24GB cards.

Formats

Mistral AI

Mistral 3 flagship MoE (41B active / 675B total) with vision encoder. FP8 on 8×H200; GGUF quant for research clusters only. Its attention is DeepSeek-style MLA, so the real KV cache is smaller than the figure below, which uses an unverified conventional layout. At 32K that is a rounding error beside ~390 GB of weights; at the full 256K window, treat the cache figure as an upper bound.

Formats

Zhipu GLM-4

Zhipu GLM-4 open 9B with 128K context, tool calling, and strong bilingual (EN/ZH) performance.

Formats

Google Gemma 3

Gemma 3 large instruct with long context and multimodal support. At Q4_K_M it needs about 18.3 GB at 4K, comfortable on a 24 GB card. The vision encoder is a separate 0.9 GB file.

Formats

DeepSeek

R1 reasoning distilled into Llama 3.1 8B. Best chain-of-thought for 8–12GB cards; huge community GGUF support.

Formats

Microsoft Phi

Microsoft Phi-4 dense 14B — strong reasoning for size. Q4 ~9GB fits 12GB cards with moderate context.

Formats

Alibaba Qwen3

Tiny Qwen3 with thinking mode. Q4 ~1.4GB — phones, NUC, and always-on local agents.

Formats

OpenAI GPT-OSS

OpenAI open-weight MoE (21B total / 3.6B active), shipped natively in MXFP4 — ~12.8GB runs on a 16GB card with no quality tax. Only 3.6B active params means CPU-offload stays usable.

Formats

OpenAI GPT-OSS

The big GPT-OSS (117B total / 5.1B active). Native MXFP4 checkpoint is ~61GB — fits one 80GB card or a 128GB unified-memory Mac. Partial offload on 24GB consumer cards is slow but works.

Formats

Zhipu GLM-4.5

Zhipu's agentic/reasoning MoE (106B total / 12B active). Q4 ~64GB — the sweet spot is a 96GB+ unified Mac or 2× 48GB cards. Strong tool-calling for its class.

Formats

Mistral Devstral

Mistral + All Hands agentic coding model on a Mistral Small 3.1 base. Built for repo-scale tool use rather than single-file completion. At Q4_K_M it needs about 15.7 GB at 4K, tight on a 16 GB card and comfortable on 20 GB.

Formats

Alibaba Qwen3-VL

Current-generation vision-language model that still fits a single 8–12GB card at Q4 (~5.9GB). The realistic multimodal option for people without a 24GB GPU — note the vision encoder adds VRAM the KV-cache math below does not model.

Formats

Alibaba Qwen3-VL

Multimodal MoE with only ~3B active parameters, so it stays responsive on Apple unified memory and survives CPU offload far better than a dense 30B. At Q4_K_M it needs about 19.9 GB at 4K, comfortable on a 24 GB card; Q3_K_M brings that to 15.1 GB. The vision encoder is a separate 1.1 GB file.

Formats

Mistral Magistral

Mistral's reasoning model on a Mistral Small 3.2 base, with reasoning wrapped in [THINK] tags. At Q4_K_M it needs about 15.7 GB at 4K, tight on a 16 GB card and comfortable on 20 GB — explicit chain-of-thought a level below the 70B-class reasoners most guides assume.

Formats

ByteDance Seed

Dense 36B with a native 512K context. The Q4_K_M file is 21.8 GB, so with cache and buffers it needs about 23.9 GB at 4K: tight on a 24 GB card, where Q3_K_M (19.5 GB) is the comfortable choice, or a 2×16GB split. The AWQ build is 21.1 GB, not the 4-bit rate the name suggests. Context is the real cost: 512K of KV cache is roughly 128GB on its own, so budget context first and weights second.

Formats

Mistral Ministral 3

Mistral ships the GGUF itself, in 3B / 8B / 14B and Instruct or Reasoning variants — the 8B is the one that fits an 8GB card at Q4 with room for real context. Plain full attention on all 34 layers, so the memory estimates below behave the way the calculator assumes. Quality loss per level has not been published.

Formats

Alibaba Qwen3.8

Hybrid attention: only 16 of its 64 layers keep a KV cache, which is why a 27B model holds a 262K context in about 16GB of cache rather than 64GB. Sizing it as a conventional stack overstates the cache fourfold — the estimates here count the 16 and match published measurements at 8K, 32K and 262K. Quality loss per level has not been published.

Formats

Zhipu GLM-4.7

Zhipu's lightweight member of GLM-4.7: 30B total, about 3B active per token, MIT licence. Its attention is MLA, which caches one compressed 576-value vector per token per layer instead of full keys and values — so at 32K context the cache is under 2 GB where a conventional 47-layer model would need several times that. llama.cpp runs it as a DeepSeek-2-style model. Sizes below come from bartowski's GGUF files; quality loss per level has not been published.

Formats

Moonshot Kimi Linear

Moonshot's hybrid linear-attention model: 49.1B parameters in total (the name rounds it to 48B), about 3B active, a 1M-token window. Only 7 of its 27 layers keep a growing cache, and those store MLA's compressed vectors, so even the full million tokens needs about 8 GB of cache — the weights, not the context, decide what it runs on. At Q4 a Mac with 48 GB or more runs it comfortably; a 32 GB card holds it only with no room to spare. Sizes below use the calculator's generic rates: no GGUF file size could be checked, and quality loss per level has not been published.

Formats

Google Gemma 4

Google's Gemma 4 mixture-of-experts model: about 25B parameters in the text model, roughly 4B active per token, a 256K window and image input. Only 5 of its 30 layers keep a cache that grows with context, and those use fewer, wider heads, so the cache stays small — about 0.9 GB at 32K and 5.3 GB at the full 256K. At Q4_K_M that is about 18.8 GB at 32K: comfortable on a 24 GB card, which can still load the whole window (tight, about 23.6 GB). A 16 GB card cannot hold it, since the Q4_K_M file alone is 17.0 GB. The vision encoder is a separate 1.2 GB file on top. Sizes come from bartowski's GGUF files, whose Q4_K_M works out at 5.40 bits per weight, well above the generic 4-bit rate; Unsloth's UD-Q4_K_M is a different file, 0.09 GB smaller. No per-level quality loss has been published.

Formats

Google Gemma 4

Google's dense Gemma 4 flagship: about 31B parameters, a 256K window and image input. Only 10 of its 60 layers cache the full context, and those use four wide KV heads, so 128K of context adds about 11 GB of cache where a conventional 60-layer design would need about 120 GB. At Q4_K_M it needs about 22.1 GB at 4K, tight on a 24 GB card; Q3_K_M is about 18.3 GB at 4K. A 32 GB card holds Q4_K_M at 32K comfortably. Google also publishes its own quantization-aware (QAT) Q4_0 build, a single 17.7 GB file: about 20.1 GB at 4K, comfortable on 24 GB up to 16K. Its 1.2 GB vision encoder is a separate file, and with it loaded even 4K is tight on 24 GB. The other levels are sized from bartowski's GGUF files, and no per-level quality loss has been published. That repository also carries two MTP draft-head files (349 MB at Q4_0, 515 MB at Q8_0) for speculative decoding; they are not the model itself and are not counted in these sizes.

Formats

Google Gemma 4

Google's on-device Gemma 4: about 4.7B parameters doing the work plus a 2.8B per-layer embedding table, a 128K window, and image and audio input. Only 4 of its 42 layers keep a cache that grows with context, so the full 128K window costs about 2.0 GB of cache. At Q4_K_M it needs about 5.3 GB at 4K, comfortable on an 8 GB card, and 7.5 GB at 128K, tight on an 8 GB card. An 8 GB Mac gives its GPU about 6 GB, so 5.3 GB at 4K is tight on an 8 GB Mac. Sizes here count the whole model in GPU memory, which is what a Mac or a full GPU load uses; llama.cpp keeps the embedding table in system RAM instead, so on a discrete card real VRAM use is lower than shown. Sizes come from unsloth's files (Q4_K_M 5.0 GB, Q8_0 8.2 GB); other publishers' Q4_K_M files are up to 0.4 GB larger. The image and audio encoder is a separate 1.0 GB file.

Formats

Google Gemma 4

The smallest Gemma 4: about 2.3B parameters doing the work plus a 2.3B per-layer embedding table, a 128K window, and image and audio input. Its cache barely grows — 3 of 35 layers track the full context, about 0.8 GB at 128K. At Q4 it is about 3.0 GB at 4K and 3.8 GB at the full window — comfortable on any card in this index and on an 8 GB Mac. Sizes count the whole model in GPU memory; llama.cpp keeps the embedding table in system RAM, so on a discrete card it needs roughly 1.3 GB less VRAM than shown at Q4. Sizes use the calculator's generic rates — no GGUF file size could be checked.

Formats

Hugging Face SmolLM

Hugging Face's fully open 3B model, with its training data and recipe published alongside the weights. It has a reasoning mode you can switch on or off, and a 64K window it was actually trained to, which reaches 128K only by YaRN extension. Four KV heads keep the cache small, so the window is affordable: at Q4_K_M it fits a 4 GB card at short context and an 8 GB card at its full trained window. That also makes it one of the few models here that is realistic on a CPU-only machine. Sizes use the calculator's generic rates, since no GGUF file size could be checked, and no per-level quality loss has been published.

Formats

Baidu ERNIE 4.5

Baidu's open mixture-of-experts model: about 21.8B parameters in total and roughly 3B active per token. Each token goes through 6 of 64 routed experts plus 2 shared ones. The total decides the memory and the active count decides the speed, so it needs the VRAM of a 20B-class model and reads about as much per token as a 3B one. At Q4_K_M it needs about 14.2 GB at 4K context, just past the comfortable line on a 16 GB card (it loads, with little margin). A 24 GB card, or a 24 GB Mac through the GPU's share of unified memory, leaves real room for context. Sizes come from the published GGUF files of the PT (instruct) checkpoint; bartowski's Q4_K_M is a different file, 0.17 GB larger than Unsloth's used here. No per-level quality loss has been published.

Formats

Allen AI OLMo

Allen AI's fully open 7B model, released with its training data, code and intermediate checkpoints. Its predecessor stopped at a 4K context; OLMo 3 reaches 64K, extended from an 8K base by YaRN. Three of every four layers use a 4K sliding window, which is what makes that affordable. It still uses full multi-head attention with no shared KV heads, so long context costs more here than on most 7B–8B models. At Q4_K_M it needs about 6.8 GB at 4K context and 10.9 GB at 32K. Without the sliding window, 32K would need about 22 GB. Sizes use the calculator's generic rates, since no GGUF file size could be checked, and no per-level quality loss has been published.

Formats

Meta Muse

Meta's first open-weight model since Llama 4, released in August under Apache 2.0 and built to run on one card. It is a dense model: a 28B text decoder that reads images through a separate vision encoder and answers in text. Three of every four layers use a 2K sliding window and the rest share just 2 KV heads, so long context is unusually cheap. At Q4_K_M it needs about 17.7 GB at 4K context and only 19.4 GB at its full 128K window, comfortable on a 24 GB card either way. Q5_K_M is tight on 24 GB, and Q8_0 wants a 32 GB card. The vision encoder (about 1.4 GB as a GGUF mmproj file) and Meta's optional speculative-decoding drafter load on top and are not included. Sizes come from the published GGUF file sizes; the Q4_K_M row is Meta's own official build. Meta's benchmark claims are its own, and no per-level quality loss has been published.

Formats

Qwen3.8

Qwen's preview of its next architecture: a 125B mixture-of-experts model with about 6B parameters active per token. Released in August, it also carries a 51B n-gram lookup table. llama.cpp reads that table lazily from RAM or disk, so it never needs VRAM. It does need disk space and is bundled inside the GGUF files, which is why they are far larger than the sizes below. Only 12 of its 48 layers keep a growing KV cache, so even the full 256K window adds about 7.5 GB. At Q4_K_M it needs about 79 GB at 4K context, comfortable on a 128 GB Mac or DGX Spark; at 256K it needs about 87 GB, still comfortable on the Spark but tight on the Mac, whose GPU gets about three quarters of its memory. A single 24 or 32 GB card runs it only with experts offloaded to system RAM. llama.cpp supports it as `qwen4exp`. Sizes use the calculator's generic rates, since no file size isolates the model from the table, and no per-level quality loss has been published.

Formats

Alibaba Qwen3.6

One of the most recommended models for a 24 GB card this year: 35.5B parameters, about 3B active per token, so it reads per token about as much as a 3B model. Only 10 of its 40 layers keep a growing KV cache, so context is cheap: at Q4_K_M it needs about 23.4 GB at 4K and 24.0 GB at 32K. On this site's rule that is tight on a 24 GB card, not comfortable, because it counts a working buffer most guides leave out. A 32 GB card, or a 48 GB Mac through the GPU's share of unified memory, has real room. It reads images through a separate vision encoder. Sizes come from bartowski's GGUF files; Unsloth's UD-Q4_K_M is a different file, 0.15 GB smaller. No per-level quality loss has been published.

Formats

Alibaba Qwen3.6

The dense model of the Qwen3.6 generation and a popular local coding model. It has the same layout as Qwen3.8 27B, the newer release: only 16 of its 64 layers keep a growing KV cache. At Q4_K_M it needs about 19.1 GB at 4K context and 21.0 GB at 32K, comfortable on a 24 GB card either way. Q5_K_M is tight on 24 GB, and a 16 GB card cannot hold it at any level listed here. Sizes come from bartowski's published GGUF files; Unsloth's builds of the same levels are about 1 GB smaller. No per-level quality loss has been published.

Formats

Alibaba Qwen3.5

The newest small Qwen and a common pick for 8 to 16 GB cards. Only 8 of its 32 layers keep a growing KV cache, so it can use a long context on modest hardware. At Q4_K_M it needs about 6.6 GB at 4K and 7.0 GB at 16K, comfortable on an 8 GB card; at 32K (7.5 GB) it is tight there and comfortable from 10 GB up. Its full 256K window needs about 15.2 GB. Q8_0 needs about 10.4 GB, comfortable from 12 GB up. Reasoning is off by default on the small Qwen3.5 sizes, and it reads images through a separate vision encoder. Sizes come from the published GGUF file sizes, and no per-level quality loss has been published.

Formats

Alibaba Qwen3.5

The 4B size of Qwen3.5, for 4 to 8 GB cards, base-model Macs and CPU-only machines. Only 8 of its 32 layers keep a growing KV cache. At Q4_K_M it needs about 3.3 GB at 4K context and 4.3 GB at 32K, so a 6 GB card is enough for long chats; its full 256K window needs about 12 GB. Reasoning is off by default on the small Qwen3.5 sizes, and it reads images through a separate vision encoder. Sizes come from the published GGUF file sizes, and no per-level quality loss has been published.

Formats

Alibaba Qwen3.5

The 2B size of Qwen3.5, small enough for almost anything: old cards, base-model Macs, mini PCs and CPU-only servers. Only 6 of its 24 layers keep a growing KV cache, with 2 KV heads each. At Q4_K_M it needs about 1.5 GB at 4K context, 1.9 GB at 32K and 4.8 GB at its full 256K window. Even Q8_0 stays around 2.2 GB at 4K. Reasoning is off by default on the small Qwen3.5 sizes. Sizes come from the published GGUF file sizes, and no per-level quality loss has been published.

Formats

Alibaba Qwen3.5

The smallest Qwen3.5, for phones, single-board computers and as a fast draft or routing model next to a bigger one. At Q4_K_M it needs about 0.6 GB at 4K context and under 1 GB at 32K. Its full 256K window needs about 3.8 GB, almost all of it cache, so very long contexts cost more than the weights here. Sizes use the calculator's generic rates, since no GGUF file size could be checked, and no per-level quality loss has been published.

Formats

Xiaomi MiMo

Xiaomi's open-weight flagship, near the top of the open-model rankings this autumn. It has about 1.02 trillion parameters, roughly 42B of them active per token, and ships under the MIT licence. The released weights are already 4-bit (MXFP4 experts), and the GGUF builds repack them rather than quantize again. At that level it needs about 604 GB at 4K context, more than any single machine in this index. The largest Mac Studio gives its GPU about 384 GB, and four 80 GB H100s hold 320 GB. Eight 80 GB cards (640 GB) are about the smallest server that holds it, and even that is tight on this site's rule. It is listed so the calculator can say so. Only 10 of its 70 layers keep a growing cache, so even 1M tokens adds only about 50 GB. llama.cpp runs it as `mimo2`, and vision loads through a separate encoder. Sizes come from a published GGUF file; no per-level quality loss has been published.

Formats

JetBrains Mellum

JetBrains' open coding model for agents and fast sub-agents, released in October under Apache 2.0. It is a 12B mixture-of-experts model with about 2.5B parameters active per token, so it generates at roughly the speed of a 3B model. At Q4_K_M it needs about 8.6 GB at 4K context and 10.5 GB even at its full 128K window, because only 7 of its 28 layers keep a growing cache. That is comfortable on a 10 or 12 GB card; an 8 GB card cannot hold it. Q8_0 fits a 16 GB card. llama.cpp supports it as `mellum`. Sizes come from JetBrains' own GGUF files; JetBrains publishes KL divergence rather than perplexity, so no quality-loss figure is shown.

Formats

IBM Granite

IBM's dense 8B reasoning model from the Granite 4.2 family (3B, 8B, 30B), released in August under Apache 2.0. Thinking is on by default; a non-thinking and a low-effort mode are chosen through the chat template. Unlike Granite 4.0's hybrid models it is a plain transformer, so llama.cpp runs it as `granite` and every one of its 40 layers keeps a full cache: about 6.5 GB at Q4_K_M with a 4K context, 11.3 GB at 32K and 27.8 GB across the whole 128K window. That makes it an 8 GB card model at short contexts and a 16 GB one for long documents. Sizes come from bartowski's GGUF files; IBM publishes no perplexity by quantization level, so no quality-loss figure is shown.

Formats

IBM Granite

The largest model in IBM's Granite 4.2 family, released in August under Apache 2.0. Like the 8B it is a plain dense transformer rather than Granite 4.0's hybrid design, so all 64 layers keep a full cache and context is not cheap: at Q4_K_M it needs about 19.6 GB at 4K, 27.3 GB at 32K and 53.7 GB across the whole 128K window. A 24 GB card runs it comfortably up to 8K and tightly at 16K; 32K needs a 32 GB card, and the full window an 80 GB card or a 96 GB Mac. Q8_0 (about 33.6 GB at 4K) wants a 48 GB card or a 64 GB Mac. Sizes come from IBM's own GGUF files; IBM publishes no perplexity by quantization level, so no quality-loss figure is shown.

Formats

IBM Granite

The smallest model in IBM's Granite 4.2 family, released in August under Apache 2.0: a dense 3.7B transformer for laptops, small cards and CPU servers. At Q4_K_M it needs about 2.7 GB at 4K context, 5.1 GB at 32K and 13.3 GB across the whole 128K window, so an 8 GB card runs it comfortably up to 32K and tightly at 64K. Q8_0 is only 4.4 GB at 4K, small enough that there is little reason to go lower on a GPU. Sizes come from IBM's own GGUF files; IBM publishes no perplexity by quantization level, so no quality-loss figure is shown.

Formats

Liquid AI LFM2.5

Liquid AI's largest LFM2.5 model, released in October: a 24B mixture-of-experts reasoning model with about 2.45B parameters active per token. It is built for on-device agents and tool calling, and always thinks before it answers. Only 10 of its 40 layers keep a growing KV cache, so long context is cheap: at Q4_K_M it needs about 15.3 GB at 4K, 15.9 GB at 32K and 17.9 GB across the whole 128K window. That is tight on a 16 GB card and comfortable on 24 GB at any context; Q8_0 (about 26.7 GB) needs a 32 GB card. Text only. Sizes come from Liquid's own GGUF files, and no per-level quality loss has been published.

Formats

Meta Muse

Meta's largest open-weight model, released in October under the Muse Model License. It is an 816B mixture-of-experts model with about 42B parameters active per token, built for coding and agent work, with image and audio input. It is a single-machine model only at its smallest build. At Q2_K (ggml-org's 304 GB file) it needs about 319 GB at 4K context and still only 327 GB at 128K, because just 16 of its 62 layers keep a growing cache — comfortable on a 512 GB Mac Studio and tight across four 80 GB H100s. The full 1M window brings it to about 389 GB. Q4_K_S (470 GB) and Q8_0 (872 GB) fit no single machine in this index. The vision and audio encoders ship as separate files of 2–4 GB and are not included. Sizes come from the published GGUF files; Meta's benchmark claims are its own and no per-level quality loss has been published. The licence is not a standard open-source one: Meta's repository is gated, so downloads need an accepted access request; products built on it must show "Built with Muse Spark Open"; and a company with over $20M in annual revenue must sign a separate agreement with Meta before offering it as an API service.

Formats

NVIDIA Nemotron

NVIDIA's Nemotron 3.5 Bolt is a 74B mixture-of-experts model with about 9B parameters active per token, built to fit a 24 GB card. It is compressed from Nemotron 3.5 Super VL and ships in one build, NVIDIA's own native 2-bit file of 22.0 GB. Only 8 of its 88 layers keep a growing cache, so context costs little: about 23.1 GB at 4K, 24.1 GB at 128K and 25.2 GB at its full 256K window. That is the 24 GB NVIDIA designed for, but over this site's comfortable line, so a 24 GB card reads as tight; a 32 GB card or a 48 GB Mac runs it comfortably. The image encoder is a separate 1.7 GB file. NVIDIA's benchmark claims are its own, and no quality loss for the 2-bit build has been published.

Formats

NVIDIA Nemotron

NVIDIA's Nemotron 3.5 Super VL is a 120B mixture-of-experts vision-language model with about 12B parameters active per token. It is built for coding, tool use and document work, with a context window of up to 1M tokens. Only 8 of its 88 layers keep a growing cache, so even the full window is cheap: at Q4_0 (ggml-org's 71.8 GB file) it needs about 75 GB at 4K and 84 GB at 1M. That runs comfortably on a 128 GB Mac or a DGX Spark 128G, and tightly on one 80 GB card. Q8_0 (128 GB) wants a 256 GB Mac. The vision encoder is a separate 1–1.7 GB file. Sizes come from the published GGUF files; NVIDIA's benchmark claims are its own and no per-level quality loss has been published.

Formats

NVIDIA Nemotron

The small member of NVIDIA's Nemotron 3.5 family: a 30B text-only mixture-of-experts model with about 3B parameters active per token. It generates at roughly the speed of a 3B model. Only 6 of its 52 layers keep a growing cache, which makes long context almost free: at Q4_0 (ggml-org's 18.9 GB file) it needs about 19.8 GB at 4K, 20.6 GB at 128K and 21.4 GB at its full 256K window. That is comfortable on a 24 GB card up to 128K and tight only at the full window; a 16 GB card cannot hold it. Q8_0 (about 35 GB) wants a 48 GB card or a 64 GB Mac. An optional 1.2 GB draft file for speculative decoding ships separately. Sizes come from the published GGUF files; NVIDIA's benchmark claims are its own and no per-level quality loss has been published.

Formats

inclusionAI Ling

inclusionAI's Ling-3.1-flash (Ant Group), released in October under MIT: a 560B mixture-of-experts reasoning model with 25B parameters active per token. It is aimed at agents, coding and long office tasks. Only 9 of its 72 layers keep a growing cache, and those store MLA's compressed vectors, so even its full 256K window adds only about 3 GB: the weights decide where it runs. At Q4_K_M it needs about 371 GB at 4K context, more than any single GPU in this index. The 512 GB Mac Studio gives its GPU about 384 GB, which holds it only tightly on this site's rule; on servers the vendor's reference setup is eight H200s or four GB300s running FP8. Q8_0 needs about 628 GB. The vendor quotes a 1M-token window, but the released config sets 262,144 positions with no scaling, so that is the limit used here. llama.cpp runs it as `bailingmoe3` from build b11571. Sizes come from a published GGUF file, whose maker reports KL divergence rather than perplexity, so no quality-loss figure is shown.

Formats

Reflection Beam

Reflection AI's first open-weight model, released in October under Apache 2.0. It is a 501B text-only mixture-of-experts model with about 23B parameters active per token, built for coding and agent work, with a 1M-token window. Only 9 of its 52 layers keep a growing cache, so context costs little: at Q2_K (DevQuasar's 182 GB file) it needs about 191 GB at 4K, 196 GB at 128K and 230 GB across the full 1M window. That is comfortable on a 512 GB Mac Studio at any context and tight on a 256 GB one up to 128K. Unsloth's dynamic 4-bit UD-Q4_K_XL file (307 GB) works out to about 321 GB at 4K, which still fits the 512 GB Mac; Q8_0 (533 GB) fits no single machine in this index. Sizes come from the published GGUF files; Reflection's benchmark claims are its own and no per-level quality loss has been published.

Formats

Google Gemma 4

Google's mid-sized Gemma 4, released in June: a dense 12B with a 256K window. It takes text, images and audio without separate encoders — the raw inputs go straight into the one decoder. Only 8 of its 48 layers cache the full context, each with a single wide KV head, so the cache stays small: about 1 GB at 32K and 4.5 GB at the full 256K. At Q4_K_M (unsloth's 7.1 GB file) that is about 8.5 GB at 32K and 10.2 GB at 128K, both comfortable on a 12 GB card; a 16 GB card holds the whole window comfortably, and an 8 GB card is tight even at 4K. bartowski's Q4_K_M is a larger file (7.7 GB), so add about half a gigabyte if you use it. Sizes come from the published GGUF files; the image and audio embedders ship as a separate mmproj file, and no per-level quality loss has been published.

Formats

Cohere North

Cohere's open coding model, released in June: a 30B mixture-of-experts with about 3B parameters active per token. It is built for code generation, agentic software work and terminal tasks, and takes text only. Only 13 of its 49 layers keep a cache that grows with context, so long sessions stay affordable: at Q4_K_M (bartowski's 18.7 GB file) it needs about 20.0 GB at 4K, 20.8 GB at 32K and 23.5 GB at 128K — comfortable on a 24 GB card up to 32K and tight at 128K. A 16 GB card needs Q3_K_M (bartowski's 14.2 GB file): about 15.3 GB at 4K, tight, and too big past 32K. The model card says 256K context while its config allows 500K; this page uses 256K. Sizes come from the published GGUF files; no per-level quality loss has been published, and Cohere's benchmark claims are its own.

Formats

OpenBMB MiniCPM-V

A mixture-of-experts vision model from OpenBMB. It reads images and video on top of a 34.7B language model with about 3B parameters active per token, the same layout as Qwen3.6 35B-A3B. Only 10 of its 40 layers keep a growing KV cache, so context is cheap. At Q4_K_S it needs about 20.9 GB at 4K, comfortable on a 24 GB card; Q4_K_M (22.2 GB) is tight there, and a 32 GB card holds Q4_K_M across the whole 256K window. The vision encoder is a separate 1.1 GB file on top of these figures, which turns Q4_K_S on 24 GB tight as well once images are loaded. llama.cpp runs the text model as `qwen35moe`. Sizes come from OpenBMB's own GGUF files, and no per-level quality loss has been published.

Formats

Xiaomi MiMo

Xiaomi's smaller MiMo-V2.6 model: 309B parameters with about 15B active per token, MIT licence. It reads text, images, video and audio. The released weights are already 4-bit on the experts (MXFP4), and ggml-org's GGUF keeps them that way: about 175 GB at 4K context. That is comfortable only on a 512 GB Mac Studio, and tight on a 256 GB one. The Q2_K build (about 132 GB) fits a 256 GB Mac comfortably and is tight on a 192 GB one. Only 9 of its 48 layers keep a growing cache, so even the full 1M window adds about 25 GB. llama.cpp runs it as `mimo2`, the same as MiMo-V2.6-Pro, and vision loads through a separate encoder. Sizes come from ggml-org's GGUF files; no per-level quality loss has been published.

Formats

MBZUAI K2 Horizon

The mixture-of-experts member of MBZUAI IFM's K2 Horizon family: 37.5B parameters with about 4B active per token, Apache 2.0, released with its training data. Every one of its 48 layers keeps an ordinary KV cache, so context is not cheap: Q4_K_M is about 24.2 GB at 4K, which is tight on a 24 GB card, and comfortable on a 32 GB card up to 16K. The 512K window would need about 129 GB. llama.cpp runs it as `k2-horizon`. Sizes come from IFM's own GGUF files (bartowski's Q4_K_M is about 0.3 GB larger); no per-level quality loss has been published.

Formats

MBZUAI K2 Horizon

The dense 32B member of MBZUAI IFM's K2 Horizon family (34.8B counted, Apache 2.0, released with its training data). All 64 layers keep an ordinary KV cache with 8 KV heads: Q4_K_M is about 23.1 GB at 4K, tight on a 24 GB card and comfortable on a 32 GB card up to 16K; each extra 32K of context adds roughly 8.8 GB. llama.cpp runs it as `k2-horizon`. Sizes come from IFM's own GGUF files (bartowski's Q4_K_M is about 1 GB larger); no per-level quality loss has been published.

Formats

MBZUAI K2 Horizon

The small dense member of MBZUAI IFM's K2 Horizon family, Apache 2.0, released with its training data. It is called 7B but holds 9.0B parameters: the 250K-token vocabulary's input and output tables add about 2B. Q4_K_M is about 6.5 GB at 4K, comfortable on an 8 GB card at that length; a 16 GB card holds it comfortably to 32K (10.8 GB), and 128K needs about 25.6 GB because every layer keeps an ordinary KV cache. llama.cpp runs it as `k2-horizon`. Sizes come from IFM's own GGUF files (bartowski's Q4_K_M is about 0.2 GB larger); no per-level quality loss has been published.

Formats

Recursive Flint

Recursive's open coding model, released on 21 October under Apache 2.0: 39.8B parameters, about 2.5B active per token. It is built for repository work such as tracing bugs and adding tests. Only 14 of its 49 layers keep a cache that grows with context, so long sessions stay cheap. On a 24 GB card the Q3_K_M build (bartowski's 19.5 GB file) needs about 20.6 GB at 4K, which is comfortable, and stays only tight up to 128K. Q4_K_M needs about 26.7 GB at 4K, so it wants a 32 GB card; ggml-org's Q4_K_M file is about 2 GB smaller than bartowski's. llama.cpp runs it as `flint`; Ollama v0.40.8 supports it on macOS only so far. Sizes come from real files; no per-level quality loss has been published.

Formats

Poolside Laguna

Poolside's agentic coding model under its OpenMDW-1.1 licence: 117.6B parameters with about 8B active per token and a 1M-token window. Only 12 of its 48 layers keep a cache that grows with context; the rest use a 512-token sliding window. At Q4_K_M (ggml-org's 67.7 GB file) it needs about 71.0 GB at 4K, which is comfortable on a 128 GB DGX Spark or Mac, and the full 1M window adds only about 53 GB more. Poolside's own Q4_K_M file is far larger (96.0 GB), so check which file you download. llama.cpp runs it as `laguna`. Sizes come from real files; no per-level quality loss has been published.

Formats

MBZUAI K2 Horizon

The smallest member of MBZUAI IFM's K2 Horizon family, Apache 2.0 and released with its training data. It is called 0.9B and holds 1.08B parameters. At Q4_K_M it needs about 1.0 GB at 4K and 2.6 GB at 32K, so it runs on almost anything, including a CPU. Every layer keeps an ordinary cache, which is why its full 128K window needs about 8.4 GB. IFM's own GGUF repo names these files 1B; the sizes here are bartowski's files, and no per-level quality loss has been published.

Formats

MBZUAI K2 Horizon

A small dense member of MBZUAI IFM's K2 Horizon family, Apache 2.0 and released with its training data. It is called 3.7B but holds 5.06B parameters: the 250K-token vocabulary's input and output tables add about 1.3B. At Q4_K_M it needs about 4.0 GB at 4K, comfortable on any card; at 32K it is about 8.3 GB, tight on an 8 GB card. Every layer keeps an ordinary cache, so 128K needs about 23.2 GB. IFM's own GGUF repo names these files 4B; the sizes here are bartowski's files, and no per-level quality loss has been published.

Formats

MBZUAI K2 Horizon

The largest K2 Horizon model from MBZUAI IFM, Apache 2.0 with its training data: 379B parameters, about 23B active per token. The parameter count is taken from its configuration. Every one of its 61 layers keeps an ordinary cache. Q2_K (bartowski's 146 GB in four parts) needs about 154 GB at 4K, comfortable on a 256 GB Mac Studio; Q4_K_M (238 GB in seven parts) needs about 250 GB and is comfortable only on a 512 GB one. IFM publishes no GGUF for this size; the sizes here are bartowski's files, and no per-level quality loss has been published.

Formats

NVIDIA Nemotron

NVIDIA's Nemotron 3.5 Bolt E4B is a small vision model built on Google's Gemma 4 E4B. It has the same 42-layer shape, about 4.7B parameters doing the work plus a 2.8B per-layer embedding table, with image input and a 64K window. Only 4 of its 42 layers keep a cache that grows with context, so long context is cheap. The one plain Q4_K_M file (LM Studio's, 6.2 GB) is much larger than the level name suggests, because the embedding table is not stored at 4 bits. It needs about 6.6 GB at 4K, comfortable on an 8 GB card, and 7.6 GB at 64K, tight on an 8 GB card. Q8_0 (an 8.0 GB file) needs about 8.5 GB at 4K and wants a 12 GB card. On a discrete card llama.cpp keeps the embedding table in system RAM, so real VRAM use is lower than shown. The vision encoder is a separate 0.6–1.0 GB file published by unsloth, and a small DFlash draft model for speculative decoding ships beside it. NVIDIA's benchmark claims are its own, and no per-level quality loss has been published.

Formats

Nanbeige

Nanbeige4.3 33B is a dense 33B model in the plain Llama layout, released under Apache 2.0. Its 512K window comes from a very large RoPE base, and every one of its 52 layers keeps a full KV cache (8 KV heads), so context is expensive: about 213 KB per token. Q4_K_M needs about 22.6 GB at 4K, tight on a 24 GB card, and 25.3 GB at 16K, which no longer fits one. Q3_K_M needs about 17.5 GB at 4K, comfortable on a 24 GB card, and 20.2 GB at 16K. A 32 GB card holds Q4_K_M comfortably to 16K. The full 512K window needs well over 100 GB of cache alone. Sizes come from bartowski's GGUF files; no per-level quality loss has been published.

Formats

XHToken Spark

Spark-X2.6 4B is a small XHToken model with a nominal 1M-token window. Only 9 of its 36 layers keep a cache that grows with context; the other 27 use a 512-token sliding window. Q4_K_M needs about 3.0 GB at 4K, comfortable on an 8 GB card, and 4.1 GB at 32K, comfortable on an 8 GB card. At 128K it needs about 7.8 GB, tight on an 8 GB card, and the full 1M window needs over 40 GB. llama.cpp runs it as spark2-5. XHToken's own Q4_K_M file is 3.5 GB because it keeps the embeddings and attention gates at 16 bits; sizes here use bartowski's standard 2.6 GB file. No per-level quality loss has been published.

Formats

Liquid AI LFM3

The smallest model of Liquid AI's LFM3 family, released on 27 October: a 1.7B text model with a 256K-token window, built for phones, laptops and small GPUs. Only 6 of its 24 layers keep a cache that grows with context; 5 use a 256-token sliding window and the other 13 are short convolutions. At Q4_K_M it needs about 1.2 GB at 4K and 2.8 GB at 128K, comfortable on an 8 GB card; the full window needs about 4.5 GB at 256K, comfortable on an 8 GB card. Q8_0 needs about 2.0 GB at 4K. Ollama runs the LFM3 family on Windows, Linux and Apple silicon since v0.40.13, and llama.cpp loads its architecture on every platform. Sizes come from Liquid's own GGUF files, and no per-level quality loss has been published.

Formats

Liquid AI LFM3

The vision-language member of Liquid AI's LFM3 family, with the same 24-layer text model shape as LFM3 1.7B. A 14-layer vision encoder ships as a separate projector file (379 MB at F16, loaded alongside the model). Only 6 of the 24 text layers keep a cache that grows with context; 5 use a 256-token sliding window and 13 are short convolutions. Its text window is 128,000 tokens, half of LFM3 1.7B's. llama.cpp and Ollama both run it.

Formats

Xiaomi MiMo

Xiaomi's MiMo-V2.6 distilled into the Qwen3.5 9B architecture, released in September, with image input through a separate vision encoder. Only 8 of its 32 layers keep a cache that grows with context. At Q4_K_M it needs about 6.3 GB at 4K and 6.7 GB at 16K, comfortable on an 8 GB card; at 32K (7.2 GB) it is tight on an 8 GB card and comfortable from 10 GB up. It needs about 10.5 GB at 128K, comfortable on a 12 GB card. Q8_0 needs about 10.1 GB at 4K, comfortable on a 12 GB card. The vision encoder adds about 0.9 GB. Sizes come from bartowski's GGUF files, and no per-level quality loss has been published.

Formats

Mistral Large

Mistral AI's open-weight flagship, released in October: a 1T mixture-of-experts model with about 52B active per token. It reads text and images (the vision encoder is a separate 3.4 GB file) and has a 1M-token window. Its compressed attention cache costs about 82 KB per token, so context is cheap next to the weights. Only the 2-bit files fit any machine in this index, a 512 GB Mac Studio: IQ2_XXS (ggml-org's 298 GB file) needs about 311 GB at 4K and 334 GB at 256K, both comfortable, and about 401 GB across the full 1M window, which is tight. Q2_K (381 GB) needs about 398 GB at 4K and is tight on that Mac up to 64K. Q4_K_M (635 GB, about 664 GB at 4K) and Q8_0 (1.1 TB) need multi-GPU servers. llama.cpp runs it since late October. Sizes come from ggml-org's GGUF files, and no per-level quality loss has been published.

Formats

Metis

The smaller of Lernex's two Metis-1.6 models, released on 1 November under Apache 2.0. It is a looped hybrid: 12 layers, mostly Mamba-2 with two attention layers and a small mixture-of-experts block on every layer, and the whole stack runs twice per token, so each token reads about 1.1B parameters' worth of weights from a 2.95B model. Each pass keeps its own attention cache, about 8 KB per token in total, which makes its 128K window cheap. At Q4_K_M it needs about 2.2 GB at 4K and 3.2 GB at 128K, comfortable on a 4 GB card. Q8_0 needs about 4.5 GB at 128K, comfortable on a 6 GB card. llama.cpp runs it since 3 November. Sizes come from Lernex's own GGUF files, and no per-level quality loss has been published.

Formats

Metis

The larger of Lernex's two Metis-1.6 models, released on 1 November under Apache 2.0. It is a looped hybrid: 20 layers, mostly Mamba-2 with three attention layers and a mixture-of-experts block on every layer, and the whole stack runs twice per token, so each token reads about 2.6B parameters' worth of weights from a 10.24B model. Each pass keeps its own attention cache, about 15 KB per token in total. IQ4_XS needs about 6.4 GB at 4K and 6.9 GB at 32K, comfortable on an 8 GB card. Q4_K_M needs about 7.1 GB at 4K, tight on an 8 GB card and comfortable from 10 GB up; at 128K it needs about 9.1 GB, comfortable on a 12 GB card. Q8_0 needs about 11.7 GB at 4K, comfortable on a 16 GB card. llama.cpp runs it since 3 November. Sizes come from Lernex's own GGUF files, and no per-level quality loss has been published.

Formats