Quant Hub

Meta Llama 3.1

Meta's flagship 8B model with 128K context. Best-in-class for local deployment.

Formats

Meta Llama 3.1

Meta's frontier 70B model. Requires 40GB+ VRAM; dual 3090 or M2 Ultra.

Formats

Meta Llama 3.2

Tiny but capable. Runs on 4GB VRAM or 8GB RAM, even on phones via llama.cpp.

Formats

Alibaba Qwen2.5

Alibaba's highly optimized 7B. Punches well above its weight, especially in coding.

Formats

Alibaba Qwen2.5

The sweet spot between performance and resource usage. 16GB VRAM with Q4.

Formats

Alibaba Qwen2.5

Near-GPT-4 reasoning on a 24GB VRAM card (Q4_K_S). Groundbreaking value.

Formats

DeepSeek

MoE architecture coding model. Active params ~2.4B, total ~16B. Exceptional code quality. Its attention is MLA: a GGUF converted by current llama.cpp (tensors attn_k_b / attn_v_b) caches about 1 GB at 32K, which is what the estimates here assume. An older conversion with a single attn_kv_b tensor caches full keys and values instead — about 8.4 GB at 32K.

Formats

Microsoft Phi

Microsoft's tiny powerhouse. Best 4B model for on-device deployment.

Formats

Mistral AI

Mistral + NVIDIA collaboration. 128K context, excellent multilingual support.

Formats

Alibaba Qwen2.5

Flagship Qwen2.5. Requires dual 4090 or A100 80G. Exceptional reasoning at scale.

Formats

DeepSeek

R1 reasoning distilled into 14B. Huge community interest; excellent chain-of-thought.

Formats

Meta Llama 3.3

Latest Meta 70B with improved multilingual. Drop-in upgrade from Llama 3.1 70B.

Formats

Mistral AI

Mistral's efficient 24B. Strong multilingual; fits on 24GB with Q4.

Formats

Alibaba Qwen2.5

Top-tier open coding model. HumanEval competitive with GPT-4o on 32B scale.

Formats

Alibaba Qwen2.5

Best 7B coding model. Ideal for local dev assistants on 8–16GB VRAM.

Formats

Alibaba Qwen2.5

Tiny Qwen2.5 for edge devices. Runs on 4GB VRAM or Raspberry Pi class hardware.

Formats

Meta Llama 3.2

Ultra-light Llama for mobile and embedded. Sub-2GB VRAM with Q4.

Formats

DeepSeek

R1 reasoning in Llama 70B architecture. Top open reasoning model for dual-GPU setups.

Formats

Mistral AI

Mistral's dedicated code model. 80+ language support, Fill-in-the-Middle capable.

Formats

Meta Llama 3.2

Multimodal Llama with image understanding. Vision encoder adds ~2GB VRAM overhead.

Formats

NousResearch

Fine-tuned Llama 3.1 8B with improved roleplay and instruction following.

Formats

IBM Granite

IBM's enterprise-grade 8B. Strong RAG and tool-use; permissive Apache 2.0 license.

Formats

Alibaba Qwen2.5

Smallest Qwen2.5. Ideal for Raspberry Pi, phones, and ultra-low-latency demos.

Formats

Alibaba Qwen2.5

Tiny Qwen with 128K context. Surprisingly capable for summarisation and chat.

Formats

Microsoft Phi

Microsoft's mid-size Phi-3. Excellent quality-per-GB on 16GB cards.

Formats

Microsoft Phi

Latest Phi mini with improved math and code. Strong 4B-class performer.

Formats

DeepSeek

MoE general model (~2.4B active). Long context and strong multilingual chat. Its attention is MLA: a GGUF converted by current llama.cpp (tensors attn_k_b / attn_v_b) caches about 1 GB at 32K, which is what the estimates here assume. An older conversion with a single attn_kv_b tensor caches full keys and values instead — about 8.4 GB at 32K.

Formats

DeepSeek

R1 reasoning in a 7B footprint. Best value for 8–12GB VRAM CoT experiments.

Formats

DeepSeek

R1 distilled to 32B. Near-frontier reasoning on a single 24GB card (Q3/Q4).

Formats

Meta Llama 3.2

Flagship multimodal Llama. Requires dual 4090 or A100; vision adds ~3GB overhead.

Formats

TII UAE

Technology Innovation Institute's latest Falcon. Good multilingual and code mix.

Formats

Alibaba Qwen3

Latest Qwen3 dense 8B with thinking mode. Strong upgrade from Qwen2.5 7B for local deploy.

Formats

Alibaba Qwen3

Qwen3 14B — best balance of reasoning and VRAM in the 2026 Qwen lineup.

Formats

Google Gemma 3

Google Gemma 3 multimodal 4B. 128K context; strong vision + text on 8GB cards.

Formats

Google Gemma 3

Mid-size Gemma 3 with vision. Fits 16GB at Q4; excellent multilingual chat.

Formats

Meta Llama 4

Meta Llama 4 Scout MoE (17B active / 109B total). Multimodal; needs ~68GB VRAM at Q4_K_M.

Formats

Meta Llama 4

Llama 4 Maverick flagship MoE (17B active / 400B total). Multi-GPU or H100 cluster territory.

Formats

Meta Llama 3.1

Meta frontier dense 405B. Q4 needs ~230GB+ VRAM; dual H100 80G or 8× consumer GPU.

Formats

Alibaba Qwen3

Qwen3 dense 32B — successor to Qwen2.5-32B with stronger reasoning and thinking mode.

Formats

Alibaba Qwen3

Qwen3 MoE with only 3B active params. Q4 ~19GB file; outperforms QwQ-32B on 16GB cards.

Formats

Alibaba Qwen3

Qwen3 flagship MoE (22B active / 235B total). Q4_K_M ~142GB; rivals DeepSeek-R1 class models.

Formats

DeepSeek

DeepSeek-V3 frontier MoE (~37B active / 671B total). MLA + FP8; multi-node GPU cluster required at Q4. MLA cache: a GGUF converted by current llama.cpp (attn_k_b / attn_v_b) holds about 2 GB at 32K, which the estimates here assume; an older conversion with a single attn_kv_b tensor caches full keys and values — about 150 GB at 32K.

Formats

DeepSeek

DeepSeek-R1 reasoning model built on V3 MoE. Chain-of-thought at frontier scale — use distill variants for local GPUs. MLA cache: a GGUF converted by current llama.cpp (attn_k_b / attn_v_b) holds about 2 GB at 32K, which the estimates here assume; an older conversion with a single attn_kv_b tensor caches full keys and values — about 150 GB at 32K.

Formats

Alibaba Qwen3

Smallest Qwen3 dense with thinking mode. Q4 ~3.2GB — ideal for 8GB GPUs and edge devices.

Formats

Alibaba Qwen3

Agentic coding MoE with 3.3B active params and 256K native context. Top open coder for 16–24GB cards.

Formats

Mistral AI

Mistral 3 flagship MoE (41B active / 675B total) with vision encoder. FP8 on 8×H200; GGUF quant for research clusters only. Its attention is DeepSeek-style MLA, so the real KV cache is smaller than the figure below, which uses an unverified conventional layout. At 32K that is a rounding error beside ~390 GB of weights; at the full 256K window, treat the cache figure as an upper bound.

Formats

Zhipu GLM-4

Zhipu GLM-4 open 9B with 128K context, tool calling, and strong bilingual (EN/ZH) performance.

Formats

Google Gemma 3

Gemma 3 large instruct with long context and multimodal support. Q4 ~16GB — dual-GPU or 24GB card with short ctx.

Formats

DeepSeek

R1 reasoning distilled into Llama 3.1 8B. Best chain-of-thought for 8–12GB cards; huge community GGUF support.

Formats

Microsoft Phi

Microsoft Phi-4 dense 14B — strong reasoning for size. Q4 ~9GB fits 12GB cards with moderate context.

Formats

Alibaba Qwen3

Tiny Qwen3 with thinking mode. Q4 ~1.4GB — phones, NUC, and always-on local agents.

Formats

OpenAI GPT-OSS

OpenAI open-weight MoE (21B total / 3.6B active), shipped natively in MXFP4 — ~12.8GB runs on a 16GB card with no quality tax. Only 3.6B active params means CPU-offload stays usable.

Formats

OpenAI GPT-OSS

The big GPT-OSS (117B total / 5.1B active). Native MXFP4 checkpoint is ~61GB — fits one 80GB card or a 128GB unified-memory Mac. Partial offload on 24GB consumer cards is slow but works.

Formats

Zhipu GLM-4.5

Zhipu's agentic/reasoning MoE (106B total / 12B active). Q4 ~64GB — the sweet spot is a 96GB+ unified Mac or 2× 48GB cards. Strong tool-calling for its class.

Formats

Mistral Devstral

Mistral + All Hands agentic coding model on a Mistral Small 3.1 base. Built for repo-scale tool use rather than single-file completion. Q4 ~14GB fits a 16GB card.

Formats

Alibaba Qwen3-VL

Current-generation vision-language model that still fits a single 8–12GB card at Q4 (~5.9GB). The realistic multimodal option for people without a 24GB GPU — note the vision encoder adds VRAM the KV-cache math below does not model.

Formats

Alibaba Qwen3-VL

Multimodal MoE with only ~3B active parameters, so it stays responsive on Apple unified memory and survives CPU offload far better than a dense 30B. Q4 ~19GB — a 24GB card holds it outright.

Formats

Mistral Magistral

Mistral's reasoning model on a Mistral Small 3.2 base, with reasoning wrapped in [THINK] tags. Q4 ~14GB puts explicit chain-of-thought on a 16GB card — the level below the 70B-class reasoners most guides assume.

Formats

ByteDance Seed

Dense 36B with a native 512K context. Q4 weights are ~22GB, which lands on a 24GB card or a 2×16GB split — but the context is the real cost: 512K of KV cache is roughly 128GB on its own, so budget context first and weights second.

Formats

Mistral Ministral 3

Mistral ships the GGUF itself, in 3B / 8B / 14B and Instruct or Reasoning variants — the 8B is the one that fits an 8GB card at Q4 with room for real context. Plain full attention on all 34 layers, so the memory estimates below behave the way the calculator assumes. Quality loss per level has not been published.

Formats

Alibaba Qwen3.8

Hybrid attention: only 16 of its 64 layers keep a KV cache, which is why a 27B model holds a 262K context in about 16GB of cache rather than 64GB. Sizing it as a conventional stack overstates the cache fourfold — the estimates here count the 16 and match published measurements at 8K, 32K and 262K. Quality loss per level has not been published.

Formats

Zhipu GLM-4.7

Zhipu's lightweight member of GLM-4.7: 30B total, about 3B active per token, MIT licence. Its attention is MLA, which caches one compressed 576-value vector per token per layer instead of full keys and values — so at 32K context the cache is under 2 GB where a conventional 47-layer model would need several times that. llama.cpp runs it as a DeepSeek-2-style model. Sizes below use the calculator's generic Q4_K_M/Q8_0 rates: no GGUF file size could be checked, and quality loss per level has not been published.

Formats

Moonshot Kimi Linear

Moonshot's hybrid linear-attention model: 48B total, 3B active, a 1M-token window. Only 7 of its 27 layers keep a growing cache, and those store MLA's compressed vectors, so even the full million tokens needs about 8 GB of cache — the weights, not the context, decide what it runs on. At Q4 a Mac with 48 GB or more runs it comfortably; a 32 GB card holds it only with no room to spare. Sizes below use the calculator's generic rates: no GGUF file size could be checked, and quality loss per level has not been published.

Formats

Google Gemma 4

Google's Gemma 4 mixture-of-experts model: about 25B parameters in the text model, roughly 4B active per token, a 256K window and image input. Only 5 of its 30 layers keep a cache that grows with context, and those use fewer, wider heads, so the cache stays small — about 0.9 GB at 32K and 5.3 GB at the full 256K. At Q4 that is about 17 GB at 32K: comfortable on a 24 GB card, which can still load the whole window (tight). A 16 GB card is just too small. Sizes use the calculator's generic rates — no GGUF file size could be checked, and no per-level quality loss has been published.

Formats

Google Gemma 4

Google's dense Gemma 4 flagship: about 31B parameters, a 256K window and image input. Only 10 of its 60 layers cache the full context, and those use four wide KV heads, so 128K of context adds about 11 GB of cache where a conventional 60-layer design would need about 120 GB. At Q4 it is about 21 GB at 4K — on a 24 GB card that is right at the edge of comfortable, and tight from 32K. A 32 GB card holds 32K comfortably. Sizes use the calculator's generic rates — no GGUF file size could be checked, and no per-level quality loss has been published.

Formats

Google Gemma 4

Google's on-device Gemma 4: about 4.7B parameters doing the work plus a 2.8B per-layer embedding table, a 128K window, and image and audio input. Only 4 of its 42 layers keep a cache that grows with context, so the full 128K window costs about 2.0 GB of cache. At Q4 it is about 4.9 GB at 4K and 7.0 GB at the full window — an 8 GB card runs it comfortably at everyday context lengths. Sizes here count the whole model in GPU memory, which is what a Mac or a full GPU load uses; llama.cpp keeps the embedding table in system RAM instead, so on a discrete card it needs roughly 1.6 GB less VRAM than shown at Q4. Sizes use the calculator's generic rates — no GGUF file size could be checked.

Formats

Google Gemma 4

The smallest Gemma 4: about 2.3B parameters doing the work plus a 2.3B per-layer embedding table, a 128K window, and image and audio input. Its cache barely grows — 3 of 35 layers track the full context, about 0.8 GB at 128K. At Q4 it is about 3.0 GB at 4K and 3.8 GB at the full window — comfortable on any card in this index and on an 8 GB Mac. Sizes count the whole model in GPU memory; llama.cpp keeps the embedding table in system RAM, so on a discrete card it needs roughly 1.3 GB less VRAM than shown at Q4. Sizes use the calculator's generic rates — no GGUF file size could be checked.

Formats

Hugging Face SmolLM

Hugging Face's fully open 3B model, with its training data and recipe published alongside the weights. It has a reasoning mode you can switch on or off, and a 64K window it was actually trained to, which reaches 128K only by YaRN extension. Four KV heads keep the cache small, so the window is affordable: at Q4_K_M it fits a 4 GB card at short context and an 8 GB card at its full trained window. That also makes it one of the few models here that is realistic on a CPU-only machine. Sizes use the calculator's generic rates, since no GGUF file size could be checked, and no per-level quality loss has been published.

Formats

Baidu ERNIE 4.5

Baidu's open mixture-of-experts model: about 21.8B parameters in total and roughly 3B active per token. Each token goes through 6 of 64 routed experts plus 2 shared ones. The total decides the memory and the active count decides the speed, so it needs the VRAM of a 20B-class model and reads about as much per token as a 3B one. At Q4_K_M it needs about 14.1 GB at 4K context, which is right at the edge of comfortable on a 16 GB card and tight by 32K. A 24 GB card, or a 24 GB Mac through the GPU's share of unified memory, leaves real room for context. Sizes use the calculator's generic rates, since no GGUF file size could be checked, and no per-level quality loss has been published.

Formats

Allen AI OLMo

Allen AI's fully open 7B model, released with its training data, code and intermediate checkpoints. Its predecessor stopped at a 4K context; OLMo 3 reaches 64K, extended from an 8K base by YaRN. Three of every four layers use a 4K sliding window, which is what makes that affordable. It still uses full multi-head attention with no shared KV heads, so long context costs more here than on most 7B–8B models. At Q4_K_M it needs about 6.8 GB at 4K context and 10.9 GB at 32K. Without the sliding window, 32K would need about 22 GB. Sizes use the calculator's generic rates, since no GGUF file size could be checked, and no per-level quality loss has been published.

Formats

Meta Muse

Meta's first open-weight model since Llama 4, released in August under Apache 2.0 and built to run on one card. It is a dense model: a 28B text decoder that reads images through a separate vision encoder and answers in text. Three of every four layers use a 2K sliding window and the rest share just 2 KV heads, so long context is unusually cheap. At Q4_K_M it needs about 17.7 GB at 4K context and only 19.4 GB at its full 128K window, comfortable on a 24 GB card either way. Q5_K_M is tight on 24 GB, and Q8_0 wants a 32 GB card. The vision encoder (about 1.4 GB as a GGUF mmproj file) and Meta's optional speculative-decoding drafter load on top and are not included. Sizes come from the published GGUF file sizes; the Q4_K_M row is Meta's own official build. Meta's benchmark claims are its own, and no per-level quality loss has been published.

Formats

Qwen3.8

Qwen's preview of its next architecture: a 125B mixture-of-experts model with about 6B parameters active per token. Released in August, it also carries a 51B n-gram lookup table. llama.cpp reads that table lazily from RAM or disk, so it never needs VRAM. It does need disk space and is bundled inside the GGUF files, which is why they are far larger than the sizes below. Only 12 of its 48 layers keep a growing KV cache, so even the full 256K window adds about 7.5 GB. At Q4_K_M it needs about 79 GB at 4K context and 87 GB at 256K, comfortable on a 128 GB Mac or DGX Spark. A single 24 or 32 GB card runs it only with experts offloaded to system RAM. llama.cpp supports it as `qwen4exp`. Sizes use the calculator's generic rates, since no file size isolates the model from the table, and no per-level quality loss has been published.

Formats

Alibaba Qwen3.6

One of the most recommended models for a 24 GB card this year: 35.6B parameters, about 3B active per token, so it reads per token about as much as a 3B model. Only 10 of its 40 layers keep a growing KV cache, so context is cheap: at Q4_K_M it needs about 23.4 GB at 4K and 24.0 GB at 32K. On this site's rule that is tight on a 24 GB card, not comfortable, because it counts a working buffer most guides leave out. A 32 GB card, or a 48 GB Mac through the GPU's share of unified memory, has real room. It reads images through a separate vision encoder. Sizes come from the published GGUF file sizes, and no per-level quality loss has been published.

Formats

Alibaba Qwen3.6

The dense model of the Qwen3.6 generation and a popular local coding model. It has the same layout as Qwen3.8 27B, the newer release: only 16 of its 64 layers keep a growing KV cache. At Q4_K_M it needs about 19.1 GB at 4K context and 21.0 GB at 32K, comfortable on a 24 GB card either way. Q5_K_M is tight on 24 GB, and a 16 GB card cannot hold it at any level listed here. Sizes come from bartowski's published GGUF files; Unsloth's builds of the same levels are about 1 GB smaller. No per-level quality loss has been published.

Formats

Alibaba Qwen3.5

The newest small Qwen and a common pick for 8 to 16 GB cards. Only 8 of its 32 layers keep a growing KV cache, so it can use a long context on modest hardware. At Q4_K_M it needs about 6.6 GB at 4K and 7.5 GB at 32K, comfortable on an 8 GB card. Its full 256K window needs about 15.2 GB. Q8_0 needs about 10.4 GB, comfortable from 12 GB up. Reasoning is off by default on the small Qwen3.5 sizes, and it reads images through a separate vision encoder. Sizes come from the published GGUF file sizes, and no per-level quality loss has been published.

Formats

Alibaba Qwen3.5

The 4B size of Qwen3.5, for 4 to 8 GB cards, base-model Macs and CPU-only machines. Only 8 of its 32 layers keep a growing KV cache. At Q4_K_M it needs about 3.3 GB at 4K context and 4.3 GB at 32K, so a 6 GB card is enough for long chats; its full 256K window needs about 12 GB. Reasoning is off by default on the small Qwen3.5 sizes, and it reads images through a separate vision encoder. Sizes come from the published GGUF file sizes, and no per-level quality loss has been published.

Formats

Alibaba Qwen3.5

The 2B size of Qwen3.5, small enough for almost anything: old cards, base-model Macs, mini PCs and CPU-only servers. Only 6 of its 24 layers keep a growing KV cache, with 2 KV heads each. At Q4_K_M it needs about 1.5 GB at 4K context, 1.9 GB at 32K and 4.8 GB at its full 256K window. Even Q8_0 stays around 2.2 GB at 4K. Reasoning is off by default on the small Qwen3.5 sizes. Sizes come from the published GGUF file sizes, and no per-level quality loss has been published.

Formats

Alibaba Qwen3.5

The smallest Qwen3.5, for phones, single-board computers and as a fast draft or routing model next to a bigger one. At Q4_K_M it needs about 0.6 GB at 4K context and under 1 GB at 32K. Its full 256K window needs about 3.8 GB, almost all of it cache, so very long contexts cost more than the weights here. Sizes use the calculator's generic rates, since no GGUF file size could be checked, and no per-level quality loss has been published.

Formats

Xiaomi MiMo

Xiaomi's open-weight flagship, near the top of the open-model rankings this autumn. It has about 1.02 trillion parameters, roughly 42B of them active per token, and ships under the MIT licence. The released weights are already 4-bit (MXFP4 experts), and the GGUF builds repack them rather than quantize again. At that level it needs about 604 GB at 4K context, more than any single machine in this index. The largest Mac Studio gives its GPU about 384 GB, and four 80 GB H100s hold 320 GB. Eight 80 GB cards (640 GB) are about the smallest server that holds it, and even that is tight on this site's rule. It is listed so the calculator can say so. Only 10 of its 70 layers keep a growing cache, so even 1M tokens adds only about 50 GB. llama.cpp runs it as `mimo2`, and vision loads through a separate encoder. Sizes come from a published GGUF file; no per-level quality loss has been published.

Formats

JetBrains Mellum

JetBrains' open coding model for agents and fast sub-agents, released in October under Apache 2.0. It is a 12B mixture-of-experts model with about 2.5B parameters active per token, so it generates at roughly the speed of a 3B model. At Q4_K_M it needs about 8.6 GB at 4K context and 10.5 GB even at its full 128K window, because only 7 of its 28 layers keep a growing cache. That is comfortable on a 10 or 12 GB card; an 8 GB card cannot hold it. Q8_0 fits a 16 GB card. llama.cpp supports it as `mellum`. Sizes come from JetBrains' own GGUF files; JetBrains publishes KL divergence rather than perplexity, so no quality-loss figure is shown.

Formats

IBM Granite

IBM's dense 8B reasoning model from the Granite 4.2 family (3B, 8B, 30B), released in August under Apache 2.0. Thinking is on by default; a non-thinking and a low-effort mode are chosen through the chat template. Unlike Granite 4.0's hybrid models it is a plain transformer, so llama.cpp runs it as `granite` and every one of its 40 layers keeps a full cache: about 6.3 GB at Q4_K_M with a 4K context, 11.1 GB at 32K and 27.6 GB across the whole 128K window. That makes it an 8 GB card model at short contexts and a 16 GB one for long documents. The Q4_K_M size comes from a published GGUF file; IBM publishes no perplexity by quantization level, so no quality-loss figure is shown.

Formats

IBM Granite

The largest model in IBM's Granite 4.2 family, released in August under Apache 2.0. Like the 8B it is a plain dense transformer rather than Granite 4.0's hybrid design, so all 64 layers keep a full cache and context is not cheap: at Q4_K_M it needs about 19.6 GB at 4K, 27.3 GB at 32K and 53.7 GB across the whole 128K window. A 24 GB card runs it comfortably up to 8K and tightly at 16K; 32K needs a 32 GB card, and the full window an 80 GB card or a 96 GB Mac. Q8_0 (about 33.6 GB at 4K) wants a 48 GB card or a 64 GB Mac. Sizes come from IBM's own GGUF files; IBM publishes no perplexity by quantization level, so no quality-loss figure is shown.

Formats

IBM Granite

The smallest model in IBM's Granite 4.2 family, released in August under Apache 2.0: a dense 3.7B transformer for laptops, small cards and CPU servers. At Q4_K_M it needs about 2.7 GB at 4K context, 5.1 GB at 32K and 13.3 GB across the whole 128K window, so an 8 GB card runs it comfortably up to 32K and tightly at 64K. Q8_0 is only 4.4 GB at 4K, small enough that there is little reason to go lower on a GPU. Sizes come from IBM's own GGUF files; IBM publishes no perplexity by quantization level, so no quality-loss figure is shown.

Formats

Liquid AI LFM2.5

Liquid AI's largest LFM2.5 model, released in October: a 24B mixture-of-experts reasoning model with about 2.45B parameters active per token. It is built for on-device agents and tool calling, and always thinks before it answers. Only 10 of its 40 layers keep a growing KV cache, so long context is cheap: at Q4_K_M it needs about 15.3 GB at 4K, 15.9 GB at 32K and 17.9 GB across the whole 128K window. That is tight on a 16 GB card and comfortable on 24 GB at any context; Q8_0 (about 26.7 GB) needs a 32 GB card. Text only. Sizes come from Liquid's own GGUF files, and no per-level quality loss has been published.

Formats

Meta Muse

Meta's largest open-weight model, released in October under the Muse Model License. It is an 816B mixture-of-experts model with about 42B parameters active per token, built for coding and agent work, with image and audio input. It is a single-machine model only at its smallest build. At Q2_K (ggml-org's 304 GB file) it needs about 319 GB at 4K context and still only 327 GB at 128K, because just 16 of its 62 layers keep a growing cache — comfortable on a 512 GB Mac Studio and tight across four 80 GB H100s. The full 1M window brings it to about 389 GB. Q4_K_S (470 GB) and Q8_0 (872 GB) fit no single machine in this index. The vision and audio encoders ship as separate files of 2–4 GB and are not included. Sizes come from the published GGUF files; Meta's benchmark claims are its own and no per-level quality loss has been published. The licence is not a standard open-source one: Meta's repository is gated, so downloads need an accepted access request; products built on it must show "Built with Muse Spark Open"; and a company with over $20M in annual revenue must sign a separate agreement with Meta before offering it as an API service.

Formats

NVIDIA Nemotron

NVIDIA's Nemotron 3.5 Bolt is a 74B mixture-of-experts model with about 9B parameters active per token, built to fit a 24 GB card. It is compressed from Nemotron 3.5 Super VL and ships in one build, NVIDIA's own native 2-bit file of 22.0 GB. Only 8 of its 88 layers keep a growing cache, so context costs little: about 23.1 GB at 4K, 24.1 GB at 128K and 25.2 GB at its full 256K window. That is the 24 GB NVIDIA designed for, but over this site's comfortable line, so a 24 GB card reads as tight; a 32 GB card or a 48 GB Mac runs it comfortably. The image encoder is a separate 1.7 GB file. NVIDIA's benchmark claims are its own, and no quality loss for the 2-bit build has been published.

Formats

NVIDIA Nemotron

NVIDIA's Nemotron 3.5 Super VL is a 120B mixture-of-experts vision-language model with about 12B parameters active per token. It is built for coding, tool use and document work, with a context window of up to 1M tokens. Only 8 of its 88 layers keep a growing cache, so even the full window is cheap: at Q4_0 (ggml-org's 71.8 GB file) it needs about 75 GB at 4K and 84 GB at 1M. That runs comfortably on a 128 GB Mac or a DGX Spark 128G, and tightly on one 80 GB card. Q8_0 (128 GB) wants a 256 GB Mac. The vision encoder is a separate 1–1.7 GB file. Sizes come from the published GGUF files; NVIDIA's benchmark claims are its own and no per-level quality loss has been published.

Formats

NVIDIA Nemotron

The small member of NVIDIA's Nemotron 3.5 family: a 30B text-only mixture-of-experts model with about 3B parameters active per token. It generates at roughly the speed of a 3B model. Only 6 of its 52 layers keep a growing cache, which makes long context almost free: at Q4_0 (ggml-org's 18.9 GB file) it needs about 19.8 GB at 4K, 20.6 GB at 128K and 21.4 GB at its full 256K window. That is comfortable on a 24 GB card up to 128K and tight only at the full window; a 16 GB card cannot hold it. Q8_0 (about 35 GB) wants a 48 GB card or a 64 GB Mac. An optional 1.2 GB draft file for speculative decoding ships separately. Sizes come from the published GGUF files; NVIDIA's benchmark claims are its own and no per-level quality loss has been published.

Formats

inclusionAI Ling

inclusionAI's Ling-3.1-flash (Ant Group), released in October under MIT: a 560B mixture-of-experts reasoning model with 25B parameters active per token. It is aimed at agents, coding and long office tasks. Only 9 of its 72 layers keep a growing cache, and those store MLA's compressed vectors, so even its full 256K window adds only about 3 GB: the weights decide where it runs. At Q4_K_M it needs about 371 GB at 4K context, more than any single GPU in this index. The 512 GB Mac Studio gives its GPU about 384 GB, which holds it only tightly on this site's rule; on servers the vendor's reference setup is eight H200s or four GB300s running FP8. Q8_0 needs about 628 GB. The vendor quotes a 1M-token window, but the released config sets 262,144 positions with no scaling, so that is the limit used here. llama.cpp runs it as `bailingmoe3` from build b11571. Sizes come from a published GGUF file, whose maker reports KL divergence rather than perplexity, so no quality-loss figure is shown.

Formats

Reflection Beam

Reflection AI's first open-weight model, released in October under Apache 2.0. It is a 501B text-only mixture-of-experts model with about 23B parameters active per token, built for coding and agent work, with a 1M-token window. Only 9 of its 52 layers keep a growing cache, so context costs little: at Q2_K (DevQuasar's 182 GB file) it needs about 191 GB at 4K, 196 GB at 128K and 230 GB across the full 1M window. That is comfortable on a 512 GB Mac Studio at any context and tight on a 256 GB one up to 128K. Unsloth's dynamic 4-bit UD-Q4_K_XL file (307 GB) works out to about 321 GB at 4K, which still fits the 512 GB Mac; Q8_0 (533 GB) fits no single machine in this index. Sizes come from the published GGUF files; Reflection's benchmark claims are its own and no per-level quality loss has been published.

Formats