Passing the recall test, failing the user: 16 memory systems, 1,000 preferences

The useful move in this design is reading the same stored fact twice. The Know test asks the agent to state a specific preference; the Act test drops it into a scenario where that preference should change the answer ^[6]. Scoring both on the same item produces four outcomes instead of two, and a retrieval benchmark can only separate remembered from not-remembered, which means the cell where the agent recites the constraint and then violates it is scored as a pass ^[17].

The paper's illustration is a user who reveals a peanut allergy while asking for help polishing an email. The memory module extracts it correctly. In a later session the agent recommends pad thai with crushed peanuts ^[14]. That failure is invisible to a recall metric, because recall is exactly the part that worked.

This matters more than a lab curiosity because of how preferences actually arrive. The authors cite a large-scale analysis of ChatGPT usage finding that most users still treat the model as a tool for refining emails, translating messages, or debugging code, so preferences surface implicitly inside task content rather than as declarations ^[13]. The benchmark embeds its 1,000 preferences at three levels of expression strength for that reason ^[3]. Weakly expressed preferences are the normal case in production, not the hard tail.

Memory architectures do help. The reported result is that they narrow the Know-Act gap, but that utilization stays weakest on health and therapy preferences ^[5], which is the inverse of where you would want the residual error. Part of the mechanism is that a memory layer is not a pure addition: distilling history into condensed context introduces its own failure points, semantic mismatch at retrieval among them ^[11]. And the field already knows models skip relevant information sitting fully in context ^[10]; adding a retrieval stage in front of that does not remove it.

Two caveats on how far this travels. What we have is one anonymised preprint with code and data behind an anonymous repository ^[16], and no per-system breakdown, so it does not license a ranking of ChatGPT's persistent memory against Claude's, or Mem0 against Letta/MemGPT, Zep, or HippoRAG, all of which the paper names as the systems being equipped with this capability ^[12]. Sixteen systems spread over five memory architectures averages 3.2 systems per architecture ^[18], which is thin for any claim about a family rather than a build.

The second caveat is the authors' own, and it cuts both ways: a good behavioural response may reflect coincidence rather than memory ^[15]. Neither half of the pair is a scorecard alone. That is the argument for pairing, and it is also why the existing evidence base is weaker than its numbers look. The benchmarks these systems post strong results on test factual recall, multi-hop reasoning, and long-range understanding ^[9], while personalization and agentic-memory work has measured recall or behaviour but not both on the same stored item ^[7], mostly in static long-context setups rather than the incremental pipelines a long-running agent actually uses ^[8]. So the closest thing to evidence that a memory layer changes what an agent does, rather than what it can repeat, is a test design that until now nobody was running.