Build1 distinct publisher3 min readPublished
A dev.to teardown ran 107 I/O-bound data engineering tasks under a sub-15-second median and a one-cent-per-task budget. What it yields is a map of where each framework's abstraction gives way once the task count climbs.
The Engineer · Build desk

Compiled by The EngineerSomething wrong?How this is made
Trace `run_pipeline` in the minimal LangGraph example. It walks the graph's topological sort and merges each node's returned dict into one state object [7]. The three nodes call a requests tool, a GPT-4-turbo completion and a Postgres insert, and each returns a dict; error handling appears in none of them [7]. A for loop over a topological sort is a scheduler in roughly the way that `while True` is a supervisor.
So the task-20 observation follows from the design, not from bad luck. Recovery logic was never in the framework, which means it goes into every node you write, along with retries and output validation [8]. The author's compensating benefit is real and worth naming: state stays transparent, so you always know which node failed and why [8]. That is a legible debugging cost. Boilerplate you can see in a diff is cheaper than state you cannot inspect.
The budget is where the design pressure actually shows. One cent per task, one to six LLM calls per task, against GPT-4-turbo or Claude 3 Opus [4][5]. At the top of the call range that leaves about $0.0017 per model call [3], while the median task payload is 100KB of text or semi-structured data [3]. The one-cent target is only reachable if most of that 100KB never reaches the model. The latency envelope is tighter still: under 15 seconds median across up to six calls is 2.5 seconds per call, including the API or database round trip that every task carries [5][4][2][4].
For the numbers to transfer, your workload has to look like this one: I/O-bound, at least one API or DB call per task, LLM-assisted rather than LLM-dominated [2]. If your agents make one long generation over a large context instead of several small calls between I/O steps, the cost curve does not carry over. The boilerplate finding does, because it is about error handling per node rather than tokens per task [8].
CrewAI's reported ceiling is the useful counterweight. The teardown says it delivers on first use, until you orchestrate beyond roughly 10 agents or tasks, at which point the task abstraction starts leaking [10]. That is a different purchase: less code now, a rewrite at a threshold you can estimate.
Two caveats on the evidence. The deadlock on task 22 and the $48 overnight bill are the author's illustration of the stakes, not measurements [12]. For scale, $48 is about 45 full passes of the whole 107-task suite at the stated one-cent target [1][2]. And the material available stops mid-sentence in CrewAI's scale gripes, with no AutoGen section and no per-framework cost or latency figures [11], despite the subtitle promising how cost and performance curves behave past 100 tasks [1]. The failure taxonomy is usable. The ranking implied by the headline is not yet checkable.
In my context I would take LangGraph's code volume, because more than a quarter of these tasks involve chaining, error recovery or fuzzy duplicate detection, which is where inspectable state pays for itself [6][5][8]. A task family that stays under ten agents and needs no cross-DAG coordination is the case where CrewAI's brevity is worth having [10][8].
Ranked by verification strength, evidence, and original report placement.
The dev.to teardown's subtitle promises a code-first teardown revealing which agent orchestration framework scales, where boilerplate becomes pain, and how cost/performance curves behave past 100+ real-world tasks.
The available text of the teardown ends mid-sentence in CrewAI's scale gripes, and contains no AutoGen section and no per-framework cost or latency figures.
The benchmark suite uses 107 production-flavored data engineering tasks: extraction (APIs, PDFs, HTML), transformation (schema normalization, deduplication, feature creation), load (Snowflake, S3, Postgres), QA (null checking, anomaly flagging) and small DAGs. Every task is I/O-bound and LLM-assisted, with at least one API or DB call per task.
Task payloads are text or semi-structured with a median size of 100KB, accessed through OpenAPI or a direct SQL/NoSQL interface.
Each task makes 1 to 6 LLM calls, using GPT-4-turbo or Claude 3 Opus.
The suite sets a hard SLO of under 15 seconds median runtime and a target cost of $0.01 per task.
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 30, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
build
94% in the demo, 11% in production: the agent gap is architectural1 distinct publisher
build
A twelve-word joke became a discipline, and one seven-step chain had no loop to remove1 distinct publisher
build
The retry sends the email twice: what agent framework comparisons leave out1 distinct publisher
build
AWS's own agent fleet guidance puts the lock-in in state, auth and telemetry, not the framework1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
One desk, no artifacts
Two kinds of evidence sit side by side here and they are not equal. The pipeline listings are self-evidencing — you can read the LangGraph nodes and see that error handling is absent. The headline finding cannot be read at all: no repository, no framework versions, no logs, no per-task failure counts stand behind 'by task 20', and the run happened on one developer's machine at an unstated time. That is why the score is low despite the specificity of the writing.
Nothing to count
Nobody reports using any of this. There is no deployment, no user count, no disclosure of the frameworks running production workloads at any company, and no sign that a second engineer has picked up the suite. The only observable artefact is the harness description itself, and a harness with no published results tells you nothing about uptake.
Curves promised, prose delivered
The overreach is in the framing, not the findings. A subtitle offering cost and performance curves past 100+ tasks sets up a data story; what follows is a specification, two listings, and bullet-point verdicts, with the $48 overnight bill working as a rhetorical stake rather than something anyone measured. Marked well short of the ceiling because the qualitative claims are unusually concrete — task 20, ten agents, three hops — and concrete claims are at least falsifiable by whoever runs the suite next.
Unknown hands
We cannot see who benefits. Nothing in this reporting says who paid for the tokens, whether any of the three projects' maintainers were involved, what the author builds for a living, or whether a product sits behind the recommendation. Absent even one of those, scoring the pull on this piece would be invention rather than assessment.
Coherent, uncorroborated, unfinished
Low, and for structural reasons rather than doubts about good faith: a single publisher, a run nobody can repeat, and a text that stops before its third framework is judged. What raises it above the floor is internal consistency — the suite spec, the code and the failure modes describe a plausible piece of work, and the specification is precise enough that a second attempt would either confirm the thresholds or expose them.