What you need first
Docker Engine, and the NVIDIA Container Toolkit specifically — an ordinary Docker install has no idea a GPU exists, and a container run without it silently falls back to the CPU with no error. Verify the toolkit works before touching Ollama at all; debugging "Ollama is slow in Docker" is much harder than debugging "does my container see the GPU".
# Verify BEFORE running Ollama:
docker run --rm --gpus=all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi
# If that fails, install the toolkit first — nothing below will work.The compose file
The GPU reservation block is what most copy-pasted examples get wrong or omit entirely. Bind the port to loopback unless you intend to expose the API — Ollama has no authentication of its own.
services:
ollama:
image: ollama/ollama
container_name: ollama
ports:
- "127.0.0.1:11434:11434"
volumes:
- ollama_data:/root/.ollama
restart: unless-stopped
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
volumes:
ollama_data:Start it and pull a model
Bring the container up, then pull a model into it — the model store lives in the named volume, so it survives `down` and `up` without a re-download. Llama 3.1 8B at Q4_K_M is 5.6 GB at 4K context, a reasonable first pull on any 8GB+ card.
docker compose up -d
docker exec ollama ollama pull llama3.1:8bCheck it actually ran on the GPU
This is the step containerisation makes easy to skip, because the container starts successfully either way — on the CPU or the GPU — and only speed tells the difference. Ask the container, not the host.
docker exec ollama ollama run llama3.1:8b "hi" >/dev/null
docker exec ollama ollama ps
# PROCESSOR should read 100% GPU
nvidia-smi --query-gpu=memory.used,utilization.gpu --format=csv -l 1What the numbers should look like
Containerisation adds no measurable overhead to inference — the GPU does the same work either way. Llama 3.1 8B at Q4_K_M is 4.6 GB of weights; on an RTX 4090 at 1,008 GB/s that puts a ceiling near 218 tok/s, which is what this index measured for this exact model and quant on that card, in Ollama, without a container. A containerised run should land at essentially the same figure.
When it does not work
`ollama ps` inside the container reports a CPU share while `nvidia-smi` on the host works fine: the `deploy.resources.reservations.devices` block is missing from the compose file, or the NVIDIA Container Toolkit is not installed — the verification command from the first section isolates which. Container starts but the toolkit check from step one already failed: fix that first; nothing about Ollama’s configuration will work around a missing toolkit. Models disappear after `docker compose down`: no named volume was used, or it was removed with `-v` — the `ollama_data` volume above is what survives a recreate. And on Windows, this requires the WSL2 backend with GPU support enabled in Docker Desktop’s settings; without that, the toolkit check fails the same way it would on a host with no toolkit at all.
Common questions
Why does Ollama in Docker use my CPU instead of my GPU?
Almost always the NVIDIA Container Toolkit is missing, or the compose file is missing its `deploy.resources.reservations.devices` block. Verify the toolkit independently first with `docker run --rm --gpus=all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi` — if that fails, the problem is the toolkit, not Ollama or your compose file.
Does running Ollama in Docker slow it down?
No, once the GPU is actually passed through. This index measured 218 tok/s for Llama 3.1 8B at Q4_K_M on an RTX 4090 running Ollama directly; a correctly configured container does the same work on the same GPU and should land at essentially the same figure. Overhead only appears when the GPU passthrough is not working and the container has silently fallen back to the CPU.
How do I keep my downloaded models after recreating the container?
Use a named volume mapped to `/root/.ollama`, as in the compose file above. That keeps the model blobs outside the container’s own lifecycle, so `docker compose down` followed by `up` does not trigger a re-download. Removing the volume explicitly with `down -v` does delete them — that is the one command to avoid.
What this guide uses
- Models
- Llama 3.1 8B Instruct8B
- Format
- GGUF
Next steps
Models covered in this guide
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
Docker Compose LLM Stack: Ollama + Open WebUI
A production-ready Docker Compose stack that gives you a local ChatGPT experience with one command.
Multi-Model API Server on RTX 4090 with vLLM
Serve multiple AWQ-quantized models with vLLM's continuous batching for production-grade throughput.
WSL2 + Ollama GPU Passthrough on Windows
Run Ollama with NVIDIA GPU acceleration inside WSL2 — the most reliable Windows path for local LLMs.