Bin-packing is the wrong scoreboard when a 70B checkpoint costs minutes

The only place this argument gets concrete is the worked example, so start there. A customer support agent calls a retrieval model on 1 GPU for 200ms, hands off to a code interpreter on CPU for 2 seconds, fans out to five parallel API calls for 500ms, then synthesizes with a reasoning model on 4 GPUs for 1 second ^[8]. That is 3.7 seconds of wall clock ^[1] against 4.2 GPU-seconds of real demand ^[2]. Hold 4 GPUs for the whole turn, which is the only way to guarantee the last step does not sit in a queue, and you have bought 14.8 GPU-seconds ^[3]. Utilization: 28 percent ^[4]. The write-up calls that posture wasteful ^[9], and by the utilization metric it is, but it is also the posture that keeps the product answering.

The alternative is worse, and arithmetic is what makes it plain. Release the GPU after step one and reacquire it for step four and you pay instance start plus model load, which the same piece puts at 30 to 90 seconds ^[5]. On a 3.7 second turn that is 8 to 24 times the length of the entire request ^[5]. So the scheduler that scores well on packing produces a support agent that stalls for a minute mid-conversation, and the scheduler that keeps the conversation moving books three idle GPU-seconds for every one it uses.

Which reframes what the proposed primitives are actually selling. Model-aware routing to a warm instance ^[10] and session affinity that pins a session to a pool so later tool calls hit the same cache ^[11] both amount to treating loaded weights as inventory rather than as slack. Speculative resource hints ^[14] are the wager on top: the agent runtime says where it is probably going, the orchestrator pre-warms shared capacity, and a wrong guess gets reassigned rather than refunded. On the training side the same logic runs slower and heavier. Checkpointing a 70B model takes minutes ^[4], so preempting a job is not a scheduling nudge, it is a multi-minute bill charged to whichever job gets moved.

Two things temper the read. The dev.to write-up credits Corey Sanders with the general case for different primitives, and carries the three-assumptions framing and every number in its own voice rather than as quotation ^[18]. And "P99 from seconds to milliseconds" ^[10] arrives with no stated model and no stated offered load, which matters more than usual because the same piece says an 80 percent GPU reading may be memory-bound rather than compute-bound ^[16]. If the percentage cannot distinguish those, it cannot verify the latency claim or the invoice.

What a buyer can put to a vendor is narrow: what does a warm-but-idle GPU cost per hour under session affinity, and who pays when a speculative pre-warm guesses wrong. The rest of the argument is downstream of those two prices.