Tokens per second is a rate, and a rate averages the two phases that bill you separately: the input you chew through before the first token, and the tokens you emit after it. The framework's own inputs concede this. It asks you to predict input and output string lengths, on the grounds that longer strings raise both GPU memory and compute demand [5]. It asks for daily active users and requests per user per day, then tells you to multiply them into a total daily workload [7].
Push that multiplication one step and you get the only volume figure the material supports: DAUs x requests per DAU per day x (ISL + OSL) tokens per day [16]. That figure sizes a token bill. It does not size a fleet, because it carries nothing about when those tokens arrive together. Concurrency is the axis it drops, and the post is explicit that high concurrency strains GPU memory and latency more than raw DAU counts do [4]. A 99th-percentile time-to-first-token target is a statement about the worst request in a hundred [3], and an average throughput number never constrains a tail.
The best-engineered input in the list is cache hit rate, defined as the share of input tokens that repeat across requests and can be served from the KV cache instead of being recomputed, which skips prefill for those tokens and cuts both TTFT and cost per request [6]. As arithmetic, prefill work per request is ISL x (1 - hit rate) [15]. It multiplies one phase and leaves decode alone, which is how it lowers the GPU capacity needed for the same traffic [6] without changing a single hardware spec. It is also a property of your product's prompts, not of your cluster.
So the conditions for a published throughput table to transfer are strict: same model, same ISL/OSL distribution, same concurrency level, same cache hit rate, and a latency figure quoted at the same percentile [2][3][6]. Drop any one and the number describes someone else's traffic. The four use case buckets in the post (chatbots and copilots, agents doing deep research and reasoning, content generation, translation apps) are there because those buckets carry different token-length and concurrency shapes [8].
Both directions of error cost money in this account: capacity ahead of the workload drops utilization and raises cost per token, capacity behind it constrains throughput and latency [11]. That asymmetry is what core-and-flex answers, with a reserved or on-prem baseline for steady state and spot or on-demand capacity for surges, launches and experiments [9], and it turns contract length into an engineering input rather than a procurement footnote [10].
Every constant here is one you measure yourself; the post supplies the shape of the calculation and none of the values [17]. Quantization, pruning and distillation show up as the levers that move the memory footprint once the traffic model is fixed [12]. The two inputs that move a GPU count most are the percentile you promise and the share of prompt tokens you already hold, and both live in your traffic rather than on a spec sheet.
