What you need first
A build of llama.cpp, and an honest question about what you are actually optimizing: token generation and prompt processing are bottlenecked by different things, and the fix for one does not touch the other. This guide covers both, separately, because conflating them is the most common mistake in CPU tuning advice.
nproc --all # logical (with hyperthreads)
lscpu | grep -E "^Core|^Socket" # physical coresWhat actually speeds up generation
Generating each token means reading every weight once, so on CPU — exactly as on a GPU — the ceiling is memory bandwidth divided by weight size. Nothing about a BLAS library changes that arithmetic. The two levers that do matter: thread count set to your physical core count (not the hyperthread-doubled figure `nproc` reports, which mostly adds contention past that point), and the quant level, since a smaller file is fewer bytes to read per token.
./build/bin/llama-server \
-m ./models/Llama-3.1-8B-Q4_K_M.gguf \
-t 8 \
-c 4096 \
--host 127.0.0.1 --port 8080
# -t 8 assumes 8 PHYSICAL cores — check with lscpu, not nprocWhat OpenBLAS actually helps
This is the correction this guide exists to make: llama.cpp’s own build documentation states plainly that BLAS acceleration helps prompt processing at batch sizes above 32, and that it does not affect generation speed at all. If your use case is chat — short prompts, long generations — OpenBLAS buys you close to nothing. If it is bulk document processing with large batched prompts, it is worth building with `-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS`.
sudo apt install -y libopenblas-dev
cmake -B build -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS
cmake --build build --config Release -j$(nproc)
# This changes prompt-processing speed on large batches.
# It will not change your tokens-per-second while chatting.Check it actually worked
Measure generation and prompt processing separately — llama.cpp prints both at the end of a run. If you built with BLAS expecting faster chat and the generation number did not move, that is not a broken build; it is the documented behaviour above.
./build/bin/llama-cli -m ./models/Llama-3.1-8B-Q4_K_M.gguf \
-p "Write a short paragraph about coffee." -n 128
# Look for two separate lines at the end:
# prompt eval time = ... tokens per second
# eval time = ... tokens per second <- this is generationWhat the numbers should look like
Llama 3.1 8B at Q4_K_M is 4.6 GB of weights. This index does not have a measured CPU throughput row, and system RAM bandwidth varies far more between machines than GPU VRAM bandwidth does — it depends on your DIMM count, speed and channel configuration, not just the CPU model — so this guide will not repeat the old version’s unsourced "~12 tok/s" and "~15 tok/s" cloud-instance figures. What you can compute for your own machine: find your system’s real memory bandwidth (from its specification, not the CPU’s marketing number) and divide by 4.6 GB for a personal ceiling.
When it does not work
Generation did not speed up after building with BLAS: expected — see above. Setting `-t` above your physical core count makes things slower, not faster: hyperthreads share execution units, and past the physical count you are adding scheduling overhead rather than throughput. Prompt processing is fast but generation is still slow: that is normal and is the bandwidth ceiling, not a misconfiguration — the only real levers left are a smaller quant or more RAM channels. And if you need consistently fast interactive chat rather than occasional CPU inference, the honest answer is a GPU, even a small one: any discrete card’s memory bandwidth is typically several times a consumer CPU’s.
Common questions
Does OpenBLAS make llama.cpp generate tokens faster?
No. llama.cpp’s own build documentation states that BLAS acceleration helps prompt processing at batch sizes above 32 and does not affect generation performance at all. If your workload is interactive chat, building with OpenBLAS will not change your tokens-per-second.
How many threads should I use for CPU inference?
Your physical core count, checked with `lscpu`, not the number `nproc` reports if hyperthreading is on — that figure is doubled and pushing thread count past the physical count typically adds contention rather than throughput. Set it with `-t`.
How fast is CPU inference compared to a GPU?
This index has no measured CPU throughput row to quote a number from, and system RAM bandwidth varies too much between machines for one figure to be meaningful. What is measurable: any discrete GPU’s memory bandwidth is typically several times higher than system RAM, and generation speed scales with that bandwidth divided by the model’s weight size — so the gap is usually large, in the GPU’s favour.
What this guide uses
- Hardware
- 32 GB RAM (CPU)32GB
- Models
- Llama 3.1 8B Instruct8B
- Format
- GGUF
Next steps
See everything that fits 💻 32 GB RAM (CPU)Reverse lookup — 32GB at 4096 context, ranked by qualityModels covered in this guide
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
Run Llama 3.1 8B on a €20/month VPS
A complete guide to running a private LLM API on a budget Linux VPS using llama.cpp server mode.
Nginx Reverse Proxy for Local LLM APIs
Put Ollama or llama.cpp behind Nginx with TLS, rate limiting, and a stable /v1 endpoint for your apps.
Running 70B on Dual RTX 3090 with llama.cpp
Split a 70B GGUF across two 24GB cards with --tensor-split — how much actually fits, and what to change when it does not.