Stop inheriting your timeout: 100 streamed requests will tell you what the budget should be

A developer writing on dev.to has published a short calibration routine for model-server timeouts, after watching a batch job fail twice under a 30-second limit he had chosen because it "felt safe" ^[1][2]. The point is not the script, which is about forty lines; it is that the number most teams are running was set once, by someone, and has survived every change to the server, the model and the workload since ^[4].

The framing is the useful part: a timeout is a latency budget, and guessing it costs you in both directions, with too short producing false failures, too long producing silent stalls, and retries amplifying either ^[3]. That reframing tells you what to measure. The guarded quantity is time-to-first-token, because if the first token never arrives the request is dead and everything after that is streaming ^[6].

The protocol is deliberately small: one endpoint, one fixed prompt, 100 sequential requests, record TTFT and total duration, compute percentiles ^[5]. A hundred samples is described as enough to see the tail without annoying the server ^[7]. The client is Python with httpx in streaming mode, because a non-streaming request hides TTFT entirely ^[8].

One caveat on the implementation as printed. The text says sequential, but the code gathers all 100 probes through asyncio.gather against a client with timeout disabled, which issues them concurrently ^[16]. That contradiction matters, because the author separately lists "sequential requests hide concurrency effects" as a limitation of the method ^[13] - the code as written does not have that limitation, and the numbers it produces are load-test numbers, not idle-path numbers ^[22].

The sample run, which the author labels illustrative rather than representative ^[10], reports TTFT of 1.2s at p50, 3.4s at p90, 5.1s at p95, 9.8s at p99, and a 22.4s maximum, with total durations of 4.8s, 9.1s, 12.6s, 21.3s and 41.7s respectively ^[9]. That is a p50-to-p99 ratio of roughly 8.2x on first token ^[18]. The recommended rule is timeout equal to p99 multiplied by 1.5, rounded up ^[11], which on those figures gives 14.7s, call it 15 ^[17]. The inherited 30 seconds was therefore twice the budget the workload could justify ^[19].

There is a second gate: if p99 exceeds 12 seconds, do not tune, investigate, because the server may be overloaded, the prompt too long, or the network the constraint ^[12]. The sample sits 2.2 seconds under that line ^[21], which is close enough that a busier hour would flip the recommendation from "set 15s" to "stop and diagnose".

Worth noting what the 30-second guess would actually have caught. No first token in the sample arrived later than 22.4s, while the slowest full response took 41.7s ^[20] - so a 30-second deadline on first token would never have fired, and a 30-second deadline on total duration would have cut a completed request. Which of those you configured determines what your logs were telling you.

The disclosure is explicit: the article was prepared as part of MonkeyCode's product outreach, and the calibration was run against MonkeyCode's free server option, with the author noting free tiers are shared resources where variance is part of the deal ^[14]. Treat the vendor's numbers as a worked example and re-run the probe against your own endpoint. The author suggests weekly, on the grounds that free servers get swapped under you ^[13].

What to watch: whether your timeout guards first token or total duration, since the sample shows those diverge by nearly 20 seconds at the maximum ^[9][20]; and whether your p99 is drifting toward the 12-second investigate threshold ^[12] rather than staying put. The method also does not cover stalls that occur after the first token arrives, which the author concedes ^[13], and a timeout is not a retry policy ^[15].