What you need first
Docker Engine with Compose v2 — `docker compose` as a subcommand, not the old `docker-compose` binary. For GPU inference you also need the NVIDIA Container Toolkit, which is the piece that lets a container see the card; without it the stack still runs, on the CPU, and nothing tells you that is what happened.
docker compose version # expect v2.x
nvidia-ctk --version # NVIDIA Container Toolkit
# Prove a container can see the GPU before going further:
docker run --rm --gpus=all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smiThe compose file
Two services: Ollama holds the models and serves the API, Open WebUI is the browser front end that talks to it. Both get named volumes, because the alternative is re-downloading tens of gigabytes the first time you recreate a container. The GPU reservation is the block people leave out.
services:
ollama:
image: ollama/ollama
container_name: ollama
ports:
- "127.0.0.1:11434:11434"
volumes:
- ollama_data:/root/.ollama
restart: unless-stopped
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
open-webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open-webui
ports:
- "127.0.0.1:3000:8080"
environment:
- OLLAMA_BASE_URL=http://ollama:11434
depends_on:
- ollama
volumes:
- webui_data:/app/backend/data
restart: unless-stopped
volumes:
ollama_data:
webui_data:Start it and pull a model
Bring the stack up, then pull a model into the running container. Llama 3.1 8B at Q4_K_M needs 5.6 GB at 4K context on this index’s numbers, so it is a safe first pull on any 8GB card. Note the port bindings above are `127.0.0.1:` prefixed — without that prefix Docker publishes to every interface, and on many setups that goes straight past the host firewall.
docker compose up -d
docker exec ollama ollama pull llama3.1:8b
# Then open http://localhost:3000Check it actually ran on the GPU
This is the step that separates a working GPU stack from one that has been quietly running on the CPU since the day it was set up. Ask the container, not the host: `ollama ps` inside it reports where each loaded model was placed.
# Load a model first, then ask where it went:
docker exec ollama ollama run llama3.1:8b "hi" >/dev/null
docker exec ollama ollama ps
# NAME SIZE PROCESSOR UNTIL
# llama3.1:8b 6.1 GB 100% GPU 4 minutes from now
# And from the host, while generating:
nvidia-smi --query-gpu=memory.used,utilization.gpu --format=csv -l 1What the numbers should look like
Containerisation costs nothing measurable at inference time — the weights are on the card either way, and the ceiling is your card’s memory bandwidth. Llama 3.1 8B at Q4_K_M is 4.6 GB of weights, so a 288 GB/s card caps generation near 62 tok/s and a 1,008 GB/s RTX 4090 near 218. What containerisation does cost is disk: the model volume grows with every pull and nothing prunes it for you.
docker exec ollama ollama list
docker system df -v | grep ollama_dataWhen it does not work
`ollama ps` says CPU inside the container while the host sees the GPU fine: the `deploy.resources.reservations.devices` block is missing or the NVIDIA Container Toolkit is not installed — the `docker run --gpus=all … nvidia-smi` check above isolates which. Models disappear after `docker compose down`: you used a bind mount that did not survive, or no volume at all; the named volume above is what keeps them. Open WebUI cannot reach Ollama: it must use the service name `http://ollama:11434`, not `localhost`, because inside the compose network `localhost` is the WebUI container itself. And if the ports are reachable from other machines, check for the `127.0.0.1:` prefix — publishing a bare port exposes it on every interface.
Common questions
How do I give a Docker container access to my NVIDIA GPU?
Install the NVIDIA Container Toolkit on the host, then declare the device in compose under `deploy.resources.reservations.devices` with `driver: nvidia` and `capabilities: [gpu]`. Verify it independently first with `docker run --rm --gpus=all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi` — if that fails, the problem is the toolkit rather than your compose file.
Why can Open WebUI not reach Ollama in the same compose file?
Because `localhost` inside a container means that container. Use the service name: `OLLAMA_BASE_URL=http://ollama:11434`. Compose puts both services on one network and resolves service names for you, so no IP address or host networking is needed.
Do I lose my downloaded models when I recreate the containers?
Only if you did not use a named volume. The `ollama_data:/root/.ollama` mapping keeps the blobs outside the container lifecycle, so `docker compose down` and `up` again costs nothing. Without it you re-download tens of gigabytes, which is the most common reason people think local models are slow to set up.
What this guide uses
- Format
- GGUF
Next steps
Models covered in this guide
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
Docker: Ollama with NVIDIA GPU Passthrough
Containerised Ollama with GPU access — isolate models, pin versions, and run alongside other services.
Mac M3 Max: The Ultimate Local LLM Setup
Maximise your Apple Silicon with Ollama. Run multiple models, set up an OpenAI-compatible API, and tune Metal GPU layers.
8GB GPU Starter Guide: 3060 / 4060 / 3070
The most common local LLM hardware tier — which models, quants, and context lengths actually fit in 8GB VRAM.