Build1 publisher3 min readPublished
A quadratic join makes prompt rebuild the top span in a profiled agent loop
A dev.to lab note clocks serialize, tool, rebuild and model as separate spans and finds assembly eating the later rounds. The join is quadratic on purpose and the model is a fixed 40ms sleep, so the harness is the transferable part.
The Engineer · Build desk
What happened
- A dev.to lab note profiles an agent loop by clocking four named spans, serialize, tool, rebuild and model, into one CSV row per round.
- The model in the harness is a forty millisecond time.sleep call, which the author describes as a ruler rather than a benchmark.
- The rebuild step re-joins the whole history once per entry, and a comment in the code states that the quadratic behaviour is deliberate.
- The author reports the language model was not the real bottleneck and that prompt rebuild ate the clock on every later round.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision Tuning inference or changing provider moves one of the four spans. A team that has not separated serialize, tool and rebuild cannot say what share of a turn it is buying back.
- capability Because the span names hold whether the model is a sleep or a network call, the same CSV lets you price your own assembly cost before you commit to a latency target with a vendor.
- contradiction The forty millisecond figure is both disclaimed as model speed and the threshold that defines the one conclusion the author kept, so the size of the finding rests on a number readers are told not to quote.
- constraint The dominant span here is removed by deleting a loop, so the result does not extend to a codebase whose prompt assembly already does one join per turn.
The result is decided by four lines inside `rebuild_prompt`. The function appends the new tool text to history, then loops over every index and reassigns `prompt = "\n".join(history[: i + 1])`, so only the final join survives and every earlier one is built and thrown away [5]. The comment above the loop says "Quadratic on purpose: a join people actually write." [6]
Put sizes on it. Each simulated tool call returns 50 file entries carrying a 2,000-character preview each, which is 100,000 characters of payload text per round [4][1]. The loop runs 12 rounds on a history seeded with two short strings [7][8], so round 12 assembles 14 entries, 12 of them payloads [2]. The prefix joins in that one round copy roughly 7.8 million characters, while the single join that actually feeds the model copies 1.2 million [3][4]. Over all 12 rounds the deliberate version moves about 36.4 million characters against 7.8 million for one join per round [5].
Two things have to hold before that crossing says anything about your agent. Your assembly path has to be doing redundant copying, and here the redundancy is a loop you can delete. Your model span also has to stay flat as the prompt grows, which is exactly what the stub does: it measures `len(prompt)` and then sleeps 40 milliseconds regardless [10]. A real endpoint prefills the prompt, so on the wire the model span and the rebuild span grow off the same history. The harness was not run against a live endpoint, though the author notes the span names stay identical when the sleep becomes a network hop [16].
Check the plotting helper before trusting the picture. It prints one `#` per millisecond of rebuild and one `.` per two milliseconds of model [14]. A 40-millisecond model therefore draws 20 dots, so the bars match when rebuild reaches 20 milliseconds, half the ruler the author tells you to watch for [2][13][6]. The CSV carries the real values, including a `prompt_chars` column you can regress `rebuild_ms` against [9].
The author is direct about the limits. "Please do not quote it as model speed," the post says of the sleep [3], and calls the whole exercise "a lab note, not a customer war story" with no production traces harvested [11]. The harness and the ASCII helper are published; measured span values from a run are not [17].
What I would keep from it is the span layout: four named intervals with one row per round, in a single CSV [1]. That is cheap to bolt onto a real loop, and it shows which of the four spans owns the round. The author's summary of the detour before he instrumented anything is the part most teams will recognise: "I spent days blaming inference like a fool." [18] The cProfile pass comes second, to name a function once the timeline has named a phase, and `json.dumps` tends to surface there first [15].
What to watch
- Whether anyone runs the same harness against a live endpoint, where prefill makes the model span grow off the same history the rebuild span does.
- Published CSV rows from a real run, which would pin down the round at which rebuild crosses the 40ms sleep.
- Whether agent frameworks expose an equivalent rebuild span, so users can see assembly cost without patching the loop themselves.