"Infinite context" is not a spec: a four-task harness for testing agent memory

A post on dev.to by tamizuddin, originally published on tamiz.pro, argues that the three phrases every agent memory vendor reaches for, "infinite context", "perfect recall" and "zero latency", collapse under real workloads, and it supplies a taxonomy plus a modular benchmark suite called MemoryBench to replace them ^[1][2][3]. That matters because none of the three phrases is falsifiable as stated, so procurement currently has nothing to hold a vendor to.

The taxonomy does most of the demolition work. Short-term memory is the model's context window: volatile, bounded by token count, and costly to extend linearly ^[4]. Long-term memory is an external store, a vector database, knowledge graph or relational store, that the agent queries to augment that window ^[5]. Once you accept the first definition, "infinite context" describes the retrieval plumbing rather than the window itself, because the window is bounded by tokens ^[19]. And the post insists the unit of measurement is the whole system: how memory is written, how it is retrieved, and how it is integrated into the reasoning loop, not retrieval accuracy alone ^[6].

The critique of published benchmarks is specific: verbatim queries that are already in the corpus and therefore guarantee high recall, corpora small enough to fit in RAM so I/O patterns never appear, no write latency measurement, and no test of how performance degrades as the store grows ^[7]. The stated alternative is end-to-end task measurement, stores grown to millions of items to simulate months of interaction, one variable changed at a time such as the embedding model, and distributions rather than averages because latency and accuracy have long tails ^[8].

MemoryBench's four tasks are where a buyer gets numbers. Factual recall runs 10,000 natural language queries against a synthetic corpus of one million user facts, reporting Recall@k, MRR and P95 retrieval latency ^[9]. The temporal task streams timestamped events and asks time-sensitive questions, scoring temporal accuracy and a staleness penalty for returning outdated facts when newer ones exist ^[10]. The write and update task replays a 90 percent read, 10 percent write trace over a simulated 30 days, reporting P95 write latency, the consistency window between a commit and its visibility to reads, and sustained writes per second ^[11]. The noise task corrupts 20 percent of the corpus with typos, paraphrases and contradictory facts, then measures recall drop against the clean corpus and a disambiguation rate that requires an agent in the loop ^[12].

That is ten named metrics ^[15]. Two caveats for anyone treating this as a purchasing checklist. None of the ten is denominated in dollars or tokens, so cost trade-offs have to be inferred from latency, throughput and the linear cost of context extension rather than read off directly ^[16]. And despite criticising public benchmarks for omitting a degradation test, the four named tasks do not include a growth-curve task of their own; the one-million-fact corpus is a scale point, not a curve ^[18]. The 10,000 queries also touch at most 1 percent of that corpus ^[17].

The reference harness is Python over ChromaDB with a deliberately generic interface, three dependencies, and a BenchmarkResult record of task, metric, value and unit ^[13]. In the published text the listing is truncated mid-definition, so treat it as a design to reimplement rather than code to clone ^[14].

Watch two numbers when you next sit across from a vendor: the consistency window and the staleness penalty ^[11][10]. Those are the ones a demo cannot show and a marketing page does not print.