Who this is for, and what actually works
A Radeon card on Linux, running GGUF through llama.cpp. Two things to settle before you start. ROCm on consumer Radeon is Linux-first: the Windows story for llama.cpp is still experimental, and the practical Windows path is Vulkan rather than HIP. And format choice is narrower than on NVIDIA — GGUF works, and vLLM ships official ROCm builds for AWQ/GPTQ serving, but EXL2 is CUDA-only and no amount of ROCm setup will change that.
Best reports: RX 7900 XTX / XT (gfx1100), W7900
Works: RX 7800 XT / 7700 XT (gfx1101), RX 6800 XT / 6900 XT (gfx1030)
Patchy: RX 6700 XT (gfx1031), older Polaris
Not supported: integrated Radeon graphicsPrerequisites
Install ROCm 6.x from AMD’s repository for your distribution, then add your user to the render and video groups and log out and back in. Missing group membership is the classic first failure: rocminfo reports no agents, and every later step looks like a build problem when it is a permissions problem. Find your card’s gfx target now — you need it for the build.
sudo usermod -aG render,video $USER # then log out and back in
rocminfo | grep -i gfx # e.g. gfx1100 for RX 7900 XTX
rocm-smi # card, VRAM, temperatureBuild llama.cpp with the HIP backend
The flag is GGML_HIP=ON. The older LLAMA_HIPBLAS name is gone, and CMake ignores unknown -D options silently — pass the old name and you get a build that compiles cleanly, runs, and is CPU-only. Set AMDGPU_TARGETS to your gfx target so the kernels are actually compiled for your card.
git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp
cmake -B build \
-DGGML_HIP=ON \
-DAMDGPU_TARGETS=gfx1100 \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=hipcc -DCMAKE_CXX_COMPILER=hipcc
cmake --build build -j$(nproc)If your card is not on the official list
ROCm refuses to initialise on gfx targets it does not officially support, even when the card is architecturally close to one that is. HSA_OVERRIDE_GFX_VERSION tells the runtime to treat it as the nearest supported target: 11.0.0 for RDNA3, 10.3.0 for RDNA2. This is a workaround, not a supported configuration — it is widely used and it can also produce wrong results or hangs on some kernels, so validate output before trusting it.
# RDNA2 card reporting gfx1031, treated as gfx1030
export HSA_OVERRIDE_GFX_VERSION=10.3.0
# RDNA3
# export HSA_OVERRIDE_GFX_VERSION=11.0.0Start the server and confirm the GPU is used
Bind to 127.0.0.1 unless you deliberately want the server on your network — llama-server has no authentication of its own. On startup the log names the backend and the number of layers offloaded; that line, not the fact that it answers, is the confirmation. A HIP build that fell back to CPU still serves tokens, slowly. rocm-smi should show VRAM held while the model is loaded.
./build/bin/llama-server \
-m ./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
-ngl 99 -c 8192 --host 127.0.0.1 --port 8080
# In the startup log, look for the ROCm device line and:
# llm_load_tensors: offloaded 33/33 layers to GPU
rocm-smi --showmemuseSizing a model for your card
A 24GB RX 7900 XTX sits in the same band as an RTX 3090 or 4090 for what fits: memory arithmetic does not care which vendor made the card, only how much VRAM it has and how large the KV cache is. Use the VRAM calculator with your Radeon selected — it is in the GPU list — rather than reading NVIDIA guidance and hoping it transfers. What does not transfer is throughput: tok/s figures on this site are measured on NVIDIA hardware and are not a prediction for ROCm.
Common problems
rocminfo finds no agents: group membership, or the kernel module did not load — check dmesg for amdgpu. hipErrorNoBinaryForGpu at load: the build did not include your gfx target; rebuild with the right AMDGPU_TARGETS, or set HSA_OVERRIDE_GFX_VERSION. Builds fine but runs on CPU: almost always the old LLAMA_HIPBLAS flag, silently ignored — check the startup log for the backend line. Hangs or garbage output after an override: the override is not a supported path; drop back to a lower context or a different quant before assuming the model is at fault.
Next steps
The models linked below are the ones this guide is written around. Open any of them with your Radeon selected in the calculator to see the context length it can actually hold.
What this guide uses
- Hardware
- Radeon RX 7900 XTX24GB
- Format
- GGUF
- Picks for it
- Best local LLM for 24GB
Next steps
See everything that fits 🔴 Radeon RX 7900 XTXReverse lookup — 24GB at 8192 context, ranked by qualityModels covered in this guide
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
Quantize Your Own Model to GGUF
Use llama.cpp's quantize tool to convert any HF model to GGUF Q4_K_M for local inference.
llama.cpp on Windows with CUDA
Build llama.cpp with NVIDIA GPU support on Windows 11 — the path of least resistance for PC gamers.
Run GPT-OSS 20B (and 120B) locally without re-quantizing
GPT-OSS ships natively in MXFP4, so the usual "download the Q4_K_M" habit makes it bigger and worse. Sizing, the right flags, and how MoE expert-offload puts the 120B on a 24GB card.