What actually fits on 48GB
A 70B at Q4_K_M is about 40.7GB of weights; with a 4K KV cache and the activation buffer the estimate lands near 46GB against 48GB of combined VRAM. That is 96% of the pair — it loads, but it is the amber band, not a comfortable fit, and it leaves nothing for a desktop session on either card. At 8K context the same model is around 47.5GB and you are running out of room to do anything else. Treat 70B Q4_K_M on 48GB as a headless configuration.
Llama 3.3 70B Q4_K_M @4K ctx ≈ 46.1 GB of 48 GB amber — loads, no margin
Llama 3.3 70B Q4_K_M @8K ctx ≈ 47.5 GB of 48 GB amber — at the ceiling
Seed-OSS 36B Q4_K_M @8K ctx ≈ 25.0 GB of 48 GB comfortable
(estimates from this site’s calculator: weights + KV cache + 10% activations)Prerequisites
Two CUDA cards visible to the driver, a llama.cpp build with CUDA enabled (GGML_CUDA=ON — the old LLAMA_CUBLAS name is silently ignored and gives you a CPU-only binary), and enough system RAM to load the file before it is distributed. The cards do not need NVLink: llama.cpp splits by layer, so the only cross-card traffic is the activations at the boundary, which PCIe handles.
nvidia-smi --query-gpu=index,name,memory.total --format=csv
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)Split across the cards
--tensor-split takes proportions, not gigabytes. "24,24" works on a matched pair only because it is the ratio 1:1 — the same as "1,1" — and writing gigabytes into it on a mismatched pair does not do what it looks like it does. On two identical 3090s, leave it at an even split and put every layer on the GPUs with -ngl 99. Bind to loopback: llama-server has no authentication of its own.
./build/bin/llama-server \
-m ./models/Llama-3.3-70B-Instruct-Q4_K_M.gguf \
--tensor-split 1,1 \
-ngl 99 -c 4096 \
--host 127.0.0.1 --port 8080Confirm both cards are carrying the model
The startup log states how many layers were offloaded; anything short of all of them means part of the model is on the CPU and generation will be slow regardless of what the cards are doing. nvidia-smi should show both GPUs holding roughly half the weights each. A single card at ~23GB while the other sits near zero means the split never took effect.
# In the startup log:
# llm_load_tensors: offloaded 81/81 layers to GPU
watch -n1 nvidia-smi --query-gpu=index,memory.used --format=csvWhen 46 of 48 GB is not enough
Three changes, in the order worth trying. Lower the context: the KV cache is the only part that scales with it, and going from 8K to 4K gives back over a gigabyte. Step down a quant level: Q3_K_M cuts the weights substantially at a real quality cost, which the model index prints per level. Or run a smaller model — a 36B at Q4_K_M is around 25GB on the same pair and leaves room to actually use the machine. What will not help is adding system RAM: once layers spill to the CPU, throughput falls off a cliff on a model this size.
Next steps
The calculator sizes one card at a time, so check a 70B against a single 24GB card to see the per-card half, then double it — and read the amber verdict as the warning it is.
What this guide uses
- Hardware
- RTX 309024GBRTX 4090
- Format
- GGUF
- Picks for it
- Best local LLM for 24GB
Next steps
See everything that fits 🟢 RTX 3090Reverse lookup — 24GB at 4096 context, ranked by qualityModels covered in this guide
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
Run Llama 3.1 8B on a €20/month VPS
A complete guide to running a private LLM API on a budget Linux VPS using llama.cpp server mode.
CPU Inference: OpenBLAS Tuning for llama.cpp
Maximize tokens/sec on a CPU-only VPS with thread count and BLAS backend tuning.
Nginx Reverse Proxy for Local LLM APIs
Put Ollama or llama.cpp behind Nginx with TLS, rate limiting, and a stable /v1 endpoint for your apps.