Loading today’s stories
Build1 distinct publisher3 min readUpdated
A preprint from ServiceNow AI Research and university co-authors reports that a plain web agent with a longer horizon matches or beats AWM, ASI and ReasoningBank, often on fewer tokens.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
Researchers at ServiceNow AI Research, with co-authors at ETS Montreal, the University of British Columbia and McGill, re-ran three popular online augmentation methods for web agents with the auxiliary token spend counted against the same budget as the actor [12][1]. Under that accounting, a vanilla actor that spends the identical budget on extra observe-and-act steps matched or surpassed all three in aggregate success rate, and often used fewer total tokens [3].
The comparison covers Agent Workflow Memory, Agent Skill Induction and ReasoningBank against what the authors call Vanilla-Increased-Budget: the augmented systems keep the default 10-step actor horizon, while the baseline converts the module budget into a longer interaction horizon [2][5]. The result holds across three WebArena domains and three models, reported as Gemini 3 Flash, GPT-5.4-mini and Qwen 3.6-27B, and the authors observe the same trend on WorkArena-L1 with Qwen 3.6-27B, which they read as evidence that the effect reaches enterprise knowledge-work tasks rather than just shopping and forum benchmarks [3][4].
The mechanism is not mysterious, and that is the point. The paper frames augmentation as an allocation problem: tokens spent inducing, retrieving, verifying or injecting reusable knowledge are tokens not spent looking at the current page, reasoning about its state, or taking one more action [7]. In the online setting the authors study, where the agent updates itself from trajectories collected during the evaluation run rather than from a skill library built beforehand, that overhead is paid on every task [1][15]. The reason this has gone unnoticed is a reporting habit: module token cost is rarely printed next to the actor's inference cost, which makes it impossible to tell whether a reported gain fits inside a fixed budget or simply bought more compute [6].
Two secondary findings matter more to anyone running these systems than the headline. First, online augmentation ties performance to task order, because accumulated memory depends on the sequence in which tasks arrive rather than the full task distribution, and the sequential dependence between tasks limits task-level parallelism [8]. That is an operational cost on top of the token cost: you lose the ability to fan work out. Second, the authors report that run-to-run variance materially affects outcomes and argue it should be treated as a core evaluation criterion for online web agents [9]. Most of the published deltas in this area are small enough that this is a substantive objection.
The authors do not claim the modules are useless. Their stated conclusion is that skills, defined here as executable code such as a Python function wrapping a reusable sequence of low-level actions, and workflow memory can help in specific domains, but that the apparent gains often disappear against a budget-matched actor [10][13]. They also offer a reason the older results looked better: prior work largely assumed base models could not finish tasks end to end, and more recent frontier models may need less external scaffolding [11].
Worth noting what the supplied excerpt does not contain: the comparisons are stated directionally, without the numeric success rates or token counts behind them [16]. Read the tables before acting on this. The practical test is cheap either way, which is the uncomfortable part: rerun your own agent with the memory stack off and the step limit raised to the same total token spend, and log both numbers side by side.
Follow any of these and your For You feed starts watching them — no settings page required.
Ranked by verification strength, evidence, and original report placement.
The paper studies online augmentation, where module overhead is paid on every task, and re-evaluates the benefits of memory, workflow and skill modules under a fixed total inference budget.
The authors compare AWM (Agent Workflow Memory), ASI (Agent Skill Induction) and ReasoningBank against a token-matched vanilla baseline that uses the same budget for additional actor steps.
Across three WebArena domains and three models, Gemini 3 Flash, GPT-5.4-mini and Qwen 3.6-27B, the vanilla baseline matches or surpasses all three augmentation methods in aggregate success rate while often using fewer total tokens.
The authors observe a similar trend on WorkArena-L1 with Qwen 3.6-27B, indicating the effect extends to enterprise knowledge-work tasks.
The augmented methods use the default 10-step actor horizon, while the budget-matched baseline, called Vanilla-Increased-Budget (Vanilla-IB), allocates the same inference budget to a longer interaction horizon.
The token cost of auxiliary modules is rarely reported next to the actor's inference cost, which makes it hard to tell whether observed gains are justified within a fixed budget.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Single self-reported preprint with a clear design but no numbers in the supplied text
The methodology is specific and falsifiable: a token-matched control against three named augmentation methods, three models, three WebArena domains plus WorkArena-L1, with an explicit call to report multi-run variance. But the supplied excerpt is abstract-and-introduction only, gives no success rates, token totals, run counts or intervals, and there is no independent replication or peer review in the cluster.
No deployment or usage evidence supplied
The only observation is the authors' own benchmark sweep inside the preprint. Nothing in the supplied material shows anyone deploying, adopting or dropping AWM, ASI, ReasoningBank or the Vanilla-IB control in production, nor any usage disclosure, so an adoption score would be invented rather than measured.
Framing runs ahead of the paper's hedged, single-preprint result
The cluster framing that memory modules 'lose' is stronger than the source's own 'matches or surpasses' and 'often uses fewer total tokens', and the paper explicitly allows that skills and workflow memory can help in specific domains and that WorkArena-L1 shows Vanilla-IB merely remaining competitive. With no numbers in the excerpt, no independent replication, and results confined to the online setting rather than offline skill libraries, the generalized claim is modestly overstated relative to the evidence.
Corporate-lab co-authorship and self-published, self-evaluated results
The work is co-authored from ServiceNow AI Research, a commercial enterprise-software AI lab, alongside three universities, and is distributed as an author-controlled preprint whose empirical claims are self-evaluated with no independent replication in the cluster. That gives the authors an interest in the negative-result framing and in the evaluation principles they advocate. Offsetting factors: the paper argues against its own field's incentives to publish scaffold gains, discloses affiliations, and presents Vanilla-IB as a control rather than a product.
Directionally plausible, weakly corroborated
Confidence is limited by a one-source cluster, self-reported and unquantified results in the supplied excerpt, and no adoption signal. It is raised by the internal coherence of the argument, the explicit control design, the breadth across three models and two benchmark suites, and the authors' own caution about run-to-run variance and domain-specific exceptions.
invest
Google says frontier models already know the facts they get wrong. That is a budget decision.1 distinct publisher
build
Qwen 3.8 27B ships thinking at maximum, and one setting stands between you and 22,000 tokens1 distinct publisher
build
A goal that writes itself into SOUL.md: agent memory is now an attack surface1 distinct publisher
build
A2A holds on the wire. ADK, Strands and Agent Framework are still not substitutable.1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.