Self-grading agents bank their own mistakes, and a second judge does not catch it

A paper posted to arxiv.org, "Memory Reward Inflation in Self-Improving LLM Agents", argues that the standard build for an agent that learns without training has a structural defect: incorrect episodes receive inflated rewards when the system grades itself, and the agent then preferentially reuses the mistakes it is most confident in ^[1][5]. If you have shipped anything with an experience bank behind it, this matters because the bad score does not average out over time; it compounds through reuse ^[6].

The architecture under examination is the common one. Each episode is stored in external memory, scored, retrieved for similar future tasks, and the new episode is written back ^[2]. The appeal is real: online adaptation without fine-tuning, personalization without retraining, reusable experience without changing the underlying model ^[16]. The authors' framing is that the stored score is a proxy reward for an implicit, non-parametric policy, and that each retrieved episode is therefore a policy-improvement step whose reliability hinges entirely on how that number was produced ^[3].

On benchmarks the number can come from ground truth: an answer checked against a gold answer, a program against tests, a database query by executing it ^[12]. In deployment, at the moment you write to memory, there is no label, so the stored reward is at best an LLM assessment ^[4]. In practice the agent or a closely related judge assigns the utility score and decides whether the episode is worth keeping, before its true success is known ^[13].

The obvious mitigation is a second model to confirm the score. According to the paper, that does not work: the confirming judge's errors remain correlated with the original self-grading bias, so it cannot identify which memories are overvalued ^[7]. The authors formalize the missing property as the Error-Independence Assumption and prove it is a necessary condition for correcting the inflation rather than merely a description of a good verifier. A usable signal has to track truth and decorrelate its error from the memory bias, and they state the recoverable payoff as a closed-form function of exactly those two quantities ^[8]. The paper names reward hacking, where a generator that also grades itself raises its self-assessed score without improving quality, as the closest relative; the difference is that here the inflated score is written to persistent storage and retrieved ^[14].

One detail undercuts a common assumption. Teams who believe their retrieval is safe because it ranks by embedding similarity rather than by stored score do not get an exemption: the paper reports the inflation compounds under plain similarity retrieval too, which it identifies as the regime a deployed agent actually uses ^[9].

The proposed remedy, an answer-free de-inflation algorithm called LUCID, is reported to raise execution accuracy on the BIRD text-to-SQL benchmark above both a Memento-style self-graded agent and a memory-less agent of identical architecture ^[10]. The abstract text supplied to us has the numeric values stripped out, so the size of that gain is unverified here ^[11]. Code, data, and per-episode memory traces are said to be published ^[15].

Worth watching: whether anyone operationalizes error independence with a signal that is structurally decorrelated from the grader, such as test execution or downstream user outcomes, rather than another model from the same family ^[8]. Also whether the similarity-retrieval result holds on tasks with no executable checker, where SQL-style verification is unavailable ^[9][12].