A memory layer beat CLAUDE.md by 22.2 points, and 54 of 72 test pairs never moved

The 22.2 points is the gap between two pass rates, and it is also the entire paired result: 17 comparisons went to RE-call, one went to the instruction file ^[6], and 16 net wins across 72 pairs is 22.2 percent ^[1]. Which leaves 54 of the 72 comparisons where both arms did the same thing, passing together or failing together ^[2]. The paired McNemar test at p = 0.000145 ^[3] says the direction is real. It does not say the effect is broad. Three quarters of a suite built around line ending handling and money rounding ^[14] could not tell the two configurations apart.

Where the effect does live is stated plainly. The eight tasks the author, writing on dev.to, labels memory-sensitive improved by 45.8 points ^[8], which over 24 cells is 11 net wins ^[3]. The other sixteen tasks carry the remaining 5 net wins across 48 cells, about 10.4 points ^[4]. One third of the suite ^[5] produces roughly two thirds of the headline, and the memory-sensitive label was applied by the person who built the memory layer. Four more tasks in that bucket and the number grows without a line of code changing.

The finding that costs nothing to act on is the one the author says he did not expect. Bare Claude Code, with no project instruction file loaded, beat the CLAUDE.md arm by 13.9 points ^[7]. That puts bare at 50.0 percent against the file's 36.1 ^[6], and it puts the memory layer only 8.3 points above running with nothing at all ^[7]. The comparison everyone will quote is the weakest arm against the winner.

Inside the mechanism, the agent searched memory in 83.3 percent of eligible sessions and reached useful context 85.0 percent of the times it searched, for 70.8 percent overall ^[10]; the two rates multiply out to the third ^[8]. So in about 29 percent of sessions the agent never got project history in front of itself ^[9], some because it did not look and some because looking did not work. That is the headroom, and it is larger than the reported gain over bare.

Cost barely constrains anything here. The complete run came to an estimated $0.4964 at captured API prices ^[11], which over 216 sessions is about a fifth of a cent each ^[10]. The four-times token multiple against the static prompt ^[12] is an argument about fleet volume, not about this experiment's budget, and the write-up sets out a per-arm cost table that the available text does not fill in ^[13].

The reusable part of this is the harness rather than the tool. A checker that only inspects the final state of a temporary repository takes the judge model out of the loop entirely ^[9], and the rig refused to count any session where the memory tools were not actually available, which the author reports never happened in this run ^[15]. Anyone with a CLAUDE.md and a suspicion that it is not paying for itself can build that much in an afternoon. On these numbers, the suspicion is worth acting on before the memory layer is.