Build1 publisher3 min readPublished
A proposed queue-cost span logs the free lane's wait as generation time
A dev.to post argues that shared inference bills you in wall-clock time, and the wrapper it prints to prove the point takes its start timestamp before the request leaves, so every successful call records a wait ratio near zero.
The Engineer · Build desk
What happened
- A dev.to post argues that the useful latency split for shared inference is queued versus generating versus retrying, not prompt versus completion.
- Its premise is that free model access and free servers absorb burst by making callers wait, since no extra workers appear on demand.
- The proposed pattern wraps whatever client you already use, assumes no vendor SDK, and appends one JSON line per attempt with failures included.
- The author labels both code blocks unexecuted examples and says the pattern is a proposal, not a production war story.
- A companion fold script groups a day of spans by job name and lane, and the post warns against starting with averages because they hide the one job that blocked a person.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- contradiction The instrument cannot show the split the post argues for: on a blocking client the queue time is recorded as generation time, so the very number you would use to abandon a free lane comes out as zero.
- decision The post's own test for choosing a lane depends on a measured wait. Adopting the wrapper unchanged gives you a column of zeros and no basis for the decision.
- cost The adoption cost lands on the client. You need one that exposes first byte, or an async submit and acknowledge, before the span has two distinct events to subtract.
- constraint Low-frequency jobs get a p95 that is really a maximum, so a nightly batch with a handful of runs will report one bad draw as its tail latency.
`span.started_at = time()` runs immediately before `send()`, carrying the code comment "move this to first byte if your client can" [6]. With a blocking HTTP client, that timestamp is the moment you submitted, not the moment a worker picked the job up. `wait_s()` subtracts `queued_at` from `started_at` [7], so what it returns is the cost of constructing a dataclass. `wait_ratio()` divides wait by wait plus generate [9]. Every successful call through the wrapper as printed reports roughly 0.000 [1].
The failure path matters more, because it is the case the post cares most about. "A timeout that never started is still a wait," the post says [11]. In `timed_call`, the except branch sets `finished_at` and records `type(exc).__name__` [10], while `started_at` was already assigned before the request went out [6]. A thirty-second timeout therefore lands as thirty seconds of generation and about zero seconds of wait [2]. The `started_at is None` fallback inside `wait_s()` [7] never fires here. It is written for an async submit-and-poll client, and the example wraps a synchronous one.
`retries` is a field on the span [5], and `timed_call` never increments it [13]. The fold script sums that column per job-and-lane group and prints it next to `ok_rate` [14]. A job that failed three times and succeeded on the fourth prints n=4, retries=0, ok_rate=0.25 [3]. The retry count has to be read off the attempt count and the success rate.
`pct()` sorts the list and indexes `round((p / 100) * (len(xs) - 1))` [15]. At eleven samples that is `round(9.5)`, which Python resolves to 10, the last index, so `p95_wait_s` returns the largest wait in the group [4]. Any group with eleven or fewer attempts in the day reports its single slowest attempt in the p95 column [4]. The post is right that averages hide the one job that blocked a person [16]; a p95 that is really a maximum hides in the other direction.
The `finally` block also writes `os.environ["LAST_WAIT_RATIO"]` [10]. That assignment mutates the environment of the Python process that made it, so a later CI step reading `$LAST_WAIT_RATIO` from the shell gets nothing, and under concurrency whichever attempt finishes last overwrites the others [5].
Two decisions in the schema are right. `lane` is a first-class field typed `"free" | "paid" | "local"` [5]. The fold can compare the same job across lanes. And the span is written in a `finally`, failures included [4]; the post is explicit that logging only successful completions hides the expensive part of the free lane [12].
For the queued-versus-generating split to be measurable, the provider has to expose two events: submission and admission to a worker. A single synchronous HTTPS request exposes one. Until a provider stamps admission in a response header, the honest version of this instrument is time-to-first-token grouped by lane. The code comment already points at that [6]. Time-to-first-token approximates queue wait; it does not measure it.
What to watch
- Measured spans from a real free-tier lane would turn the post's premise into evidence.
- A provider stamping queue admission in a response header would make wait_s measurable without an async client.
- An async submit-and-poll variant would exercise the started_at-is-None branch the current example never reaches.