How fast will it run on your card?

Why memory bandwidth sets the limit

To generate one token, a dense model reads every one of its weights from memory once. So tokens per second can never exceed `bandwidth ÷ weight bytes`: an RTX 4090 moves about 1 TB/s, and a model whose weights take 10 GB cannot pass about 100 tokens per second on it, however fast the compute is. That is why a smaller quant level is faster as well as smaller, and why a card with more bandwidth but the same memory runs the same model faster. A mixture-of-experts model reads only the experts each token is routed to, so its bound is computed over the active parameters and is much higher.

Why the estimate sits below the bound

No runtime reaches its card's peak bandwidth: kernels, drivers and per-token overhead all cost something, and how much differs by card. So the estimate starts from what this card actually did — its tg128 result for Llama 2 7B Q4_0 on llama.cpp's community scoreboards — and scales it by how many more or fewer bytes the chosen model reads per token. Checked against the dense GGUF runs in this site's benchmark table, real speeds landed between 0.99× and 1.31× of the estimate, which is the range printed beside it. Outside dense 7–33B GGUF models that check does not apply, so those rows show the bound alone.

Common questions

Why do some rows show only an upper bound?

The estimate is calibrated on llama.cpp runs of GGUF files from dense 7–33B models. A row whose recommended level is AWQ or GPTQ (run by vLLM), a mixture-of-experts model, or a model outside that size range has not been checked against a real run, so it gets the bandwidth bound only — a printed guess would look as reliable as the checked figures and would not be.

Does a long prompt or a long conversation slow it down?

Yes. These figures are for generating with a nearly empty context, which is what tg128 measures. As the context grows each token also reads more of the KV cache, so generation slows, and a long prompt adds a wait before the first token appears. The VRAM calculator shows that wait separately, from each card's measured prompt-processing speed.

Do two cards make it twice as fast?

Not with the default setup. llama.cpp and Ollama split a model across cards by layers, and the cards take turns on every token, so the speed bound stays that of one card; a second card buys capacity — a bigger model or a longer context — not speed. Tensor-parallel serving in vLLM can scale speed, at the cost of a fast link between the cards.