Build1 publisher3 min readPublished
The flowchart test sorts most agent projects back into ordinary code
Anthropic's December 2024 guidance and three agent benchmarks converge on an awkward result, because the workloads whose steps cannot be enumerated in advance are also the ones where measured agent completion is lowest.
The Engineer · Build desk
What happened
- Anthropic's December 2024 engineering guidance separates workflows, where models and tools run through predefined code paths, from agents, where the model dynamically directs its own process.
- Sierra's tau-bench introduced the pass^k metric in June 2024, and GPT-4o solved about 61% of retail customer-service tasks once but only about 25% on all eight attempts.
- Gartner predicted in June 2025 that over 40% of agentic AI projects would be canceled by the end of 2027.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision A team that can finish the flowchart has already settled its architecture, and the only open question left is which four or five points need natural language turned into structured data.
- cost By the post's estimate, every runtime rediscovery of a flowchart the team already owns costs about three cents, and it recurs on each request while a coded branch decides for nothing.
- constraint Consistency, not average accuracy, becomes the binding limit on customer-facing work, because eight identical requests are all handled correctly together only about a quarter of the time.
- exposure An agent placed on customer records inherits the near-zero confidentiality awareness CRMArena-Pro reports, so the component with the widest tool access is the one least likely to withhold data.
Multiplicative reliability is the part of this that does not turn on anyone's judgement. Take 95% success per step, which the dev.to post calls generous and puts at roughly where good tool-calling sits, and a twenty-step chain completes correctly 36% of the time [5]. Shorten it and the same rate gives 51% at thirteen steps, 49% at fourteen [1]. The post's rule of thumb, avoid agents past roughly ten sequential steps [6], sits just inside that crossover.
The tau-bench numbers say something the averages hide. Sierra's pass^k metric, published in June 2024, asks whether the agent solves the same task on all k attempts [11]. GPT-4o came in at about 61% pass@1 on retail customer-service tasks and roughly 25% pass^8 [12]. If attempts failed independently, eight consecutive successes at 61% would land 1.9% of the time [2]. The measured 25% is about thirteen times that [3], so the failures are correlated: a band of tasks the model gets right every time, and a band it gets right sometimes.
CRMArena-Pro puts a number on the enumerability question itself. Salesforce scored leading agents at 58% on single-turn tasks and 35% once the task required multiple turns, across 4,280 queries and nineteen expert-validated business tasks [9]. That is a fall of 23 points, about 40% of the single-turn score [4]. Multi-turn is the case where the next step depends on what the previous step returned. The post names that condition as the only good reason to build an agent [3], and it is where the measured score is worst.
That changes what the flowchart test is good for. Drawing the diagram tells a team when to write ordinary code [16]. Whether the tasks failing the test are ready to hand to a model is a separate question. Carnegie Mellon's TheAgentCompany gave agents 175 real professional tasks in a simulated software company, and the best model of December 2024 completed 24.0% autonomously [7]; the post reports the leaderboard has since crawled into the low thirties [8], under ten points in eighteen months [5].
The post's author is direct about why the loop gets built anyway: "nobody has ever been given a budget line item for a for-loop" [14]. Anthropic's engineering guidance, published the same month as TheAgentCompany, recommends finding the simplest solution possible and only increasing complexity when needed [2]. The post calls that "a vendor telling you to buy less of the product" [17]. On its author's own count, the enumerability rule collapses about 80% of the agent projects they have seen into ordinary software [4]. Gartner predicted in June 2025 that over 40% of agentic AI projects would be canceled by the end of 2027 [13].
What to watch
- Whether TheAgentCompany leaderboard clears 50% autonomous completion.
- Whether any model vendor publishes pass^k alongside pass@1 in its own reporting rather than leaving it to Sierra.
- Whether Gartner's 40% cancellation forecast gets tracked against actual project cancellations before end of 2027.