Why does Ollama use an hf.co/ tag instead of a short one?

Ollama library tags like `qwen2.5:7b` are curated and cannot be derived from a model name — a guessed one simply 404s. Pulling the GGUF straight from Hugging Face works for every mapped model and pins the exact quant level. If the model has a library tag you prefer, it works too, but it picks the quant for you.

What is -ngl and what should I set it to?

The number of transformer layers offloaded to the GPU. `99` means "all of them", which is what you want whenever the model fits. Lower it to split a model between VRAM and system RAM; throughput drops sharply once any layer lives in RAM, so use the VRAM calculator first to see whether you need to.

The download command matched nothing. Why?

`huggingface-cli download --include` exits 0 when its pattern matches no file, so a wrong filename looks like a successful download of nothing. Check that the repo actually publishes that quant level — smaller models often skip Q8_0, and very large ones ship sharded files whose names carry an `-00001-of-0000N` suffix.

Can I run these commands on AMD or Apple silicon?

The llama.cpp and Ollama paths, yes — pick the matching environment and the build flags change to Metal or ROCm. **ExLlamaV2 is CUDA-only.** vLLM is not: it ships official ROCm builds, so it runs on supported Radeon and Instinct cards, though kernel coverage for AWQ and GPTQ there is narrower than on CUDA and moves between releases — check the vLLM installation docs for your card before committing to it. Neither runs on Apple silicon.

$ CLI Gen

CLI Script Generator

Generate ready-to-run commands in seconds — no parameter memorisation required

Framework

Environment

Model

Quantization

Run Options

GPU Layers (n_gpu_layers)

Context Length

CPU Threads

Server Port

Run Options — API Key (optional)

Select framework, environment and model above

Where the identifiers come from

A model's display name is not an identifier. "Llama 3.1 8B Instruct" is not a Hugging Face repo, not a GGUF filename and not an Ollama tag, and a command built from it fails in a way that looks like it should have worked. The generator takes the real repo from the site's repo map and derives the GGUF filename from it, because that is the actual convention: bartowski/Meta-Llama-3.1-8B-Instruct-GGUF ships Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf.

Why some commands show a placeholder

vLLM serves FP16, AWQ or GPTQ weights — not the GGUF repos this site maps — and EXL2 conversions are published per model by different people. Neither is derivable from what the index knows, so those commands emit a visible <hf-repo-id> rather than a plausible guess. An obvious placeholder costs you one lookup; a wrong repo costs you a download and a confusing error.

Build flags that silently do nothing

llama.cpp renamed its CMake options from LLAMA_* to GGML_*. CMake does not error on an unknown option — it defines an unused variable and carries on — so -DLLAMA_CUDA=ON produces a build that compiles, runs, and is CPU-only. If a build of yours is inexplicably slow, check the flag names before anything else. The commands here use the current names.

Common questions

Why does Ollama use an hf.co/ tag instead of a short one?: Ollama library tags like qwen2.5:7b are curated and cannot be derived from a model name — a guessed one simply 404s. Pulling the GGUF straight from Hugging Face works for every mapped model and pins the exact quant level. If the model has a library tag you prefer, it works too, but it picks the quant for you.
What is -ngl and what should I set it to?: The number of transformer layers offloaded to the GPU. 99 means "all of them", which is what you want whenever the model fits. Lower it to split a model between VRAM and system RAM; throughput drops sharply once any layer lives in RAM, so use the VRAM calculator first to see whether you need to.
The download command matched nothing. Why?: huggingface-cli download --include exits 0 when its pattern matches no file, so a wrong filename looks like a successful download of nothing. Check that the repo actually publishes that quant level — smaller models often skip Q8_0, and very large ones ship sharded files whose names carry an -00001-of-0000N suffix.
Can I run these commands on AMD or Apple silicon?: The llama.cpp and Ollama paths, yes — pick the matching environment and the build flags change to Metal or ROCm. ExLlamaV2 is CUDA-only. vLLM is not: it ships official ROCm builds, so it runs on supported Radeon and Instinct cards, though kernel coverage for AWQ and GPTQ there is narrower than on CUDA and moves between releases — check the vLLM installation docs for your card before committing to it. Neither runs on Apple silicon.