Which cards this covers
Four cards in this index have 24GB: the RTX 4090, the RTX 3090, the Radeon RX 7900 XTX and the older Tesla P40. They fit exactly the same models, because only capacity decides what fits, but they do not run them at the same speed: generation is limited by memory bandwidth, which ranges from 346 GB/s on the P40 to 1,008 GB/s on the 4090. The RX 7900 XTX runs GGUF through ROCm or Vulkan; the AMD guide covers that setup. A 24GB Mac is a different case: macOS lets the GPU use about three quarters of it, so it fits fewer models than these cards.
Card Bandwidth ceiling for Qwen3.6 27B Q4_K_M
RTX 4090 1,008 GB/s 59 tok/s
RX 7900 XTX 960 GB/s 56 tok/s
RTX 3090 936 GB/s 55 tok/s
Tesla P40 346 GB/s 20 tok/s
Ceilings = bandwidth ÷ weight file size. Not measured here; real speed is lower.What fits, and where the 30B class gets tight
88 of the 117 models in this index fit a 24GB card comfortably at 4K context, and 93 load at all. Up to about 27B the question is only how long a context you want. Above that, many of the newest models land just over this site’s comfortable line, which leaves room for the model but not for your desktop. Qwen3.6 27B at Q4_K_M with a 32K context is about 21.0 GB, still comfortable. Qwen3.6 35B-A3B at Q4_K_M with a 4K context is about 23.4 GB, which is tight. K2 Horizon 32B at Q4_K_M with a 4K context is about 23.1 GB, also tight. Qwen3 32B at Q4_K_M with an 8K context is about 22.9 GB, tight, and at 32K it no longer fits.
GGUF (level) 8K 32K 64K 128K
Gemma 4 26B-A4B (Q4_K_M) 18.3 GB 18.8 GB 19.5 GB 20.9 GB
Muse Glimmer 30B (Q4_K_M) 17.7 GB 18.1 GB 18.5 GB 19.4 GB
Nemotron 3.5 Lightning (Q4_0) 19.8 GB 20.0 GB 20.2 GB 20.6 GB
Qwen3.6 27B (Q4_K_M) 19.3 GB 21.0 GB 23.2 GB ~ 27.6 GB ✗
Qwen3.8 27B (Q4_K_M) 17.8 GB 19.4 GB 21.6 GB ~ 26.0 GB ✗
North Mini Code 30B-A3B (Q4_K_M) 20.2 GB 20.8 GB 21.7 GB ~ 23.5 GB ~
Qwen3-Coder 30B-A3B (Q4_K_M) 20.1 GB 22.6 GB ~ 25.9 GB ✗
Gemma 4 31B (Q4_0, Google QAT) 20.4 GB 22.5 GB ~ 25.2 GB ✗
Qwen3.6 35B-A3B (Q4_K_M) 23.5 GB ~ 24.0 GB ~ 24.7 GB ~ 26.1 GB ✗
Qwen3 32B (Q4_K_M)* 22.9 GB ~ 29.6 GB ✗
K2 Horizon 32B (Q4_K_M) 24.2 GB ~ 30.8 GB ✗
~ = tight (88–105% of 24 GB) ✗ = does not fit * window ends at 40,960 tokensLong context: pick the architecture, not the parameter count
The table above splits into two groups. Models whose every layer keeps a growing cache, such as Qwen3 32B and K2 Horizon 32B, add several GB for every extra 32K of context. Models that keep a growing cache on only some layers barely move: the rest use a short sliding window or a fixed-size state. Gemma 4 26B-A4B at Q4_K_M with a 128K context is about 20.9 GB, comfortable. Nemotron 3.5 Lightning 30B-A3B at Q4_0 with a 128K context is about 20.6 GB. Muse Glimmer 30B at Q4_K_M with a 128K context is about 19.4 GB. If you want a whole repository or a long document in the window on one 24GB card, start with these rather than the dense 32B models.
For code
Two coding models in this index are built for this card size. North Mini Code 1.0 30B-A3B at Q4_K_M with a 32K context is about 20.8 GB, comfortable, and stays only tight up to 128K, because 13 of its 49 layers keep a growing cache. Qwen3-Coder 30B-A3B at Q4_K_M with an 8K context is about 20.1 GB, comfortable; with a 32K context it is about 22.6 GB, tight. Both are mixture-of-experts models with about 3B parameters active per token, so they generate much faster than a dense 30B. This site runs no coding benchmarks, so it does not say which writes better code.
Run it
With Ollama, pull the exact quant straight from a GGUF repo; at this size the difference between Q4_K_M and Q5_K_M is often the difference between comfortable and tight. With llama.cpp, pass -ngl all and an explicit -c. Recent builds fit the model to the card automatically and keep about 1 GiB free, so a model near the limit silently loses layers to the CPU; asking for every layer makes a setting that does not fit fail at load instead. Bind to 127.0.0.1 unless you mean to serve your network.
ollama run hf.co/bartowski/Qwen_Qwen3.6-27B-GGUF:Q4_K_M
/set parameter num_ctx 32768
ollama ps # PROCESSOR should read 100% GPU
# or llama.cpp:
hf download bartowski/Qwen_Qwen3.6-27B-GGUF --include "Qwen_Qwen3.6-27B-Q4_K_M.gguf" --local-dir ./models
./build/bin/llama-server -m ./models/Qwen_Qwen3.6-27B-Q4_K_M.gguf \
-ngl all -c 32768 --host 127.0.0.1 --port 8080
# load_tensors: offloaded 65/65 layers to GPUCheck it is really on the GPU
Any one of these is enough: ollama ps shows 100% GPU; llama.cpp’s startup log reports every layer offloaded; nvidia-smi shows the memory held while the model is loaded. A tight model is where the slow-but-working case shows up most: on Windows, the NVIDIA control panel’s System Memory Fallback is on by default and spills VRAM into system RAM instead of failing, so a model that is a few hundred MB over turns into a model that crawls. Set CUDA Sysmem Fallback Policy to prefer no fallback while you are sizing things.
nvidia-smi --query-gpu=name,memory.used,memory.total --format=csv
# measure speed on your own card:
./build/bin/llama-bench -m ./models/Qwen_Qwen3.6-27B-Q4_K_M.gguf -ngl 99 # pp512 / tg128When it does not work
On a 24GB card most problems come from a model that was only ever tight.
Loads, then runs out of memory partway through a long answer
→ the KV cache grows with the conversation; lower -c / num_ctx
Loads, but generates at a few tok/s
→ layers on the CPU: check ollama ps / the load_tensors line
→ Windows: System Memory Fallback is hiding an out-of-memory
A 35B model "fits" in the calculator but not in practice
→ it is tight: close the browser / other GPU apps, or drop one quant level
Want 70B
→ does not fit one 24 GB card at Q4_K_M; see the dual-GPU guideCommon questions
Can a 24GB GPU run a 32B model?
At Q4_K_M only with a short context, and only tight. This site sizes Qwen3 32B at Q4_K_M at about 22.9 GB with an 8K context, past its comfortable line, and with a 32K context it no longer fits. Q3_K_M brings it to about 17.7 GB at 4K. If you want a dense model of about that size with a real context, Qwen3.6 27B at Q4_K_M with a 32K context is about 21.0 GB and stays comfortable.
RTX 3090 or RTX 4090 for local LLMs?
They fit the same models: 88 of the 117 models in this index comfortably at 4K context on either card. The difference is bandwidth, 936 against 1,008 GB/s, which puts the generation ceiling for Qwen3.6 27B at Q4_K_M at about 55 tok/s on the 3090 and 59 on the 4090. For running models, as opposed to training them, a used 3090 gives up very little.
Can a 24GB GPU run a 70B model?
Not on the card. This site sizes Llama 3.3 70B at Q4_K_M at about 46.1 GB with a 4K context, nearly twice what the card holds. Two 24GB cards, or offloading most of the model to system RAM at a large speed cost, are the realistic routes; the dual-GPU guide covers the first.
What this guide uses
Next steps
See everything that fits 🟢 RTX 4090Reverse lookup — 24GB at 32768 context, ranked by qualityModels covered in this guide
- Qwen3.6 27B27B · per-quant VRAM & speed
- Gemma 4 26B-A4B IT26B-A4B · per-quant VRAM & speed
- Qwen3-Coder 30B-A3B Instruct30B-A3B · per-quant VRAM & speed
- North Mini Code 1.0 30B-A3B30B-A3B · per-quant VRAM & speed
- Nemotron 3.5 Lightning 30B-A3B30B-A3B · per-quant VRAM & speed
- Qwen3.6 35B-A3B35B-A3B · per-quant VRAM & speed
- Gemma 4 31B IT31B · per-quant VRAM & speed
- Qwen3 32B Instruct32B · per-quant VRAM & speed
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
RTX 5090: What 32GB Actually Buys for Local LLMs
What fits on an RTX 5090 that does not fit on a 24GB card: not many more models, but much longer context and higher speed ceilings. Sizes from the calculator, setup for Blackwell.
What to Run on a 12GB GPU (RTX 3060 12G, 4070, 5070)
Which models fit a 12GB NVIDIA card at 8K–32K context, where 14B stops fitting, how to run them with Ollama or llama.cpp, and how to prove they are on the GPU.
Run GPT-OSS 20B (and 120B) locally without re-quantizing
GPT-OSS ships natively in MXFP4, so the usual "download the Q4_K_M" habit makes it bigger and worse. Sizing, the right flags, and how MoE expert-offload puts the 120B on a 24GB card.