Pick two machines and one quant. Text streams only when this site has a measurement or a range scaled from that card. A bandwidth ceiling is shown as a ceiling, not played back as a speed.
RTX 4060 Ti 16G
50–66 tok/s
Scaled from this card's measured Llama 2 7B Q4_0 generation speed by the bytes read per token. The range is the checked error, 0.99–1.31.
Each token reads 4.62 GB. The file is 4.62 GB. KV cache at this context: 0.02 GB.
Time to first token 0.04 s
Run
RTX 4090
148 tok/s
Measured here. Prompt 128, generation 128, one sequence.
Each token reads 4.62 GB. The file is 4.62 GB. KV cache at this context: 0.02 GB.
Time to first token 0.01 s
Run
What is allowed to move the text
Three labels, and only the first two stream. Measured is a row in this site's benchmark matrix for that exact card, model and quant (prompt 128, generation 128, one sequence). Derived is a range: the card's own Llama 2 7B Q4_0 generation speed, scaled by how many bytes this quant reads per token, with the checked error 0.99–1.31. Text streams at the low end of that range. Ceiling is memory bandwidth divided by the bytes read per token — for a mixture-of-experts model, only the active parameters, not the whole file. It is an upper bound. Similar MoE models measured on an RTX 4090 landed near a quarter of that bound, so a ceiling never streams.
Context changes the wait, not the range
Generation rate stays put when you change context, because decode is bound by reading the weights. What changes is the KV cache and the time to first token, which is this card's measured Llama 2 7B prompt speed scaled by the model's compute. A card with no published prefill row shows no wait. If the estimate exceeds the memory this machine can use — about 75% of unified memory on a Mac, the whole pool on a discrete card — streaming stops. Partial offload to system RAM is not modelled here.
Common questions
Why does a 30B mixture-of-experts model not play on a 24 GB card?
Unless that exact card has a measured row, the only number we will stand behind is the bandwidth ceiling on the active parameters. Playing that number would look several times too fast. The RTX 4090 row for Qwen3 30B-A3B Q4_K_M is a real measurement, so that one card does stream.
Can I share what I am looking at?
Yes. Both cards, the model, the quant and the context length are in the URL.