Token-matched, agent memory modules lose to simply giving the actor more steps

Researchers at ServiceNow AI Research, with co-authors at ETS Montreal, the University of British Columbia and McGill, re-ran three popular online augmentation methods for web agents with the auxiliary token spend counted against the same budget as the actor ^[12][1]. Under that accounting, a vanilla actor that spends the identical budget on extra observe-and-act steps matched or surpassed all three in aggregate success rate, and often used fewer total tokens ^[3].

The comparison covers Agent Workflow Memory, Agent Skill Induction and ReasoningBank against what the authors call Vanilla-Increased-Budget: the augmented systems keep the default 10-step actor horizon, while the baseline converts the module budget into a longer interaction horizon ^[2][5]. The result holds across three WebArena domains and three models, reported as Gemini 3 Flash, GPT-5.4-mini and Qwen 3.6-27B, and the authors observe the same trend on WorkArena-L1 with Qwen 3.6-27B, which they read as evidence that the effect reaches enterprise knowledge-work tasks rather than just shopping and forum benchmarks ^[3][4].

The mechanism is not mysterious, and that is the point. The paper frames augmentation as an allocation problem: tokens spent inducing, retrieving, verifying or injecting reusable knowledge are tokens not spent looking at the current page, reasoning about its state, or taking one more action ^[7]. In the online setting the authors study, where the agent updates itself from trajectories collected during the evaluation run rather than from a skill library built beforehand, that overhead is paid on every task ^[1][15]. The reason this has gone unnoticed is a reporting habit: module token cost is rarely printed next to the actor's inference cost, which makes it impossible to tell whether a reported gain fits inside a fixed budget or simply bought more compute ^[6].

Two secondary findings matter more to anyone running these systems than the headline. First, online augmentation ties performance to task order, because accumulated memory depends on the sequence in which tasks arrive rather than the full task distribution, and the sequential dependence between tasks limits task-level parallelism ^[8]. That is an operational cost on top of the token cost: you lose the ability to fan work out. Second, the authors report that run-to-run variance materially affects outcomes and argue it should be treated as a core evaluation criterion for online web agents ^[9]. Most of the published deltas in this area are small enough that this is a substantive objection.

The authors do not claim the modules are useless. Their stated conclusion is that skills, defined here as executable code such as a Python function wrapping a reusable sequence of low-level actions, and workflow memory can help in specific domains, but that the apparent gains often disappear against a budget-matched actor ^[10][13]. They also offer a reason the older results looked better: prior work largely assumed base models could not finish tasks end to end, and more recent frontier models may need less external scaffolding ^[11].

Worth noting what the supplied excerpt does not contain: the comparisons are stated directionally, without the numeric success rates or token counts behind them ^[16]. Read the tables before acting on this. The practical test is cheap either way, which is the uncomfortable part: rerun your own agent with the memory stack off and the step limit raised to the same total token spend, and log both numbers side by side.