AdvancedEdge / Local 11 min read

Running 70B on Dual RTX 3090 with llama.cpp

Tensor-split across two 24GB cards to run Llama 3.1 70B or Qwen2.5 72B at Q4.

Verified stack

2× RTX 3090 24G · llama.cpp CUDA · --tensor-split · 70B Q4_K_M

Last verified 2026-07-22

70BMulti-GPUllama.cpptensor-split

Tensor split

Use --tensor-split to distribute layers across GPUs. Q4_K_M 70B needs ~44GB weights — tight but workable on 48GB total.

bash

./build/bin/llama-server \
  -m ./models/Llama-3.1-70B-Q4_K_M.gguf \
  --tensor-split 24,24 \
  -c 4096 -ngl 99 \
  --host 0.0.0.0 --port 8080

Related guides

AdvancedEdge / Local12 min read

Quantize Your Own Model to GGUF

Use llama.cpp's quantize tool to convert any HF model to GGUF Q4_K_M for local inference.

AdvancedEdge / Local12 min read

AMD GPU + llama.cpp via ROCm (Quick Start)

Run GGUF models on Radeon RX 7900 / 6800 series with llama.cpp HIP backend — what works and what does not.

BeginnerEdge / Local9 min read

llama.cpp on Windows with CUDA

Build llama.cpp with NVIDIA GPU support on Windows 11 — the path of least resistance for PC gamers.

Deployment guides are educational. Each model is subject to its own license — read the official Hugging Face model card before downloading or deploying.