45 or 793 tok/s: the same model, and only one of those numbers sizes your box

A dev.to guide on tokens-per-second benchmarking lands on one pair of numbers that should end a lot of procurement arguments: the same model can measure 45 tok/s or 793 tok/s depending on how you test it ^[1]. Most consumer benchmarks report tok/s at one user, one request ^[2], which makes them a fine description of you chatting privately with your own model ^[3] and a poor basis for sizing anything that a team will hit.

The mechanism behind the spread is batching. A naive server finishes one request before starting the next; a batched server fills idle GPU cycles with other requests ^[5]. The guide credits vLLM's continuous batching plus PagedAttention with letting one GPU absorb far more total traffic than a server that processes requests one at a time ^[6]. At a single request, the difference barely exists: Ollama and vLLM are close, with Ollama slightly ahead on some hardware ^[7].

The numbers come from a Red Hat benchmark pattern on an A100 40GB running Llama 3.1 8B, where vLLM peaked around 793 tok/s of combined throughput against about 41 tok/s for Ollama, a roughly 19x gap that appeared only as concurrency climbed from 1 to 256 users ^[8]. Ollama stayed behind even when told to run 32 parallel workers ^[9]. The author is explicit that the multiplier is not a constant and varies by model, GPU and workload ^[10].

The part that matters for a capacity plan is that aggregate throughput is not a promise to any individual. If that 793 tok/s peak was measured at 256 concurrent users, the per-user share works out to roughly 3.1 tok/s ^[1], and the same guide describes 5 tok/s as feeling like a 1998 internet connection ^[12] while 50 tok/s outruns human reading speed at roughly 100 to 150 words per second ^[11]. Run the sizing arithmetic the other way and it gets uncomfortable: holding 50 tok/s each for 20 simultaneous users requires about 1,000 tok/s of aggregate output, which is above the 793 tok/s that this configuration peaked at ^[2]. A headline throughput figure tells you the machine is busy, not that anyone is happy.

That is the throughput-versus-latency split, and the two pull in opposite directions: batching more requests raises total tokens per second but can stretch how long any one request waits before it starts producing ^[13]. Which is why serving guides track P99, the wait seen by the slowest one percent of requests ^[14]. In the same Red Hat test, vLLM held an 80 ms P99 under concurrency while Ollama's tail latency reached 673 ms ^[15], about 8.4x worse ^[3].

What to watch: any tok/s figure quoted without a concurrency level is unusable, so ask what load produced it, and ask for P99 at the concurrency you actually expect rather than at the point where throughput peaks. Two more things belong in the same conversation. Tokenization is not standard, and the word "calculator" can be one token or three depending on the tokenizer, which is why an identical prompt costs different amounts across providers ^[16]. And the number that decides your architecture is total tokens across all users per second ^[4], which is the number single-user reviews never publish.