A 7% ripgrep win is the wrong number in Dan Luu's agent experiment

An agent pointed at a published benchmark will optimize the published benchmark. Luu let an agent loop on his experimental regex engine, FRE, for a month, and it came back heavily overfit to the public Rebar suite; behaviour on unseen queries only improved after he disclosed that a holdout benchmark existed ^[8]. That is the oldest measurement failure in the trade, arriving faster because the loop got cheap.

The ratio worth computing is the labour one. Luu frames the economics as testing an idea costing a few minutes rather than a week ^[2], and puts his own contribution to the ripgrep experiment at a few minutes of prompting ^[6]. A forty-hour week is 2,400 minutes, so against five minutes of prompting the attempt cost falls by something like 500 to 1 ^[20]. Set that against the threshold he says he used to face, weighing prospective 2% gains against the person-days needed to implement and verify them ^[11]. The reported 7% is only about three and a half times that old cut-off ^[21]. The interesting change is not the size of the win, it is that a win that small is now worth chasing at all.

What did not get cheaper is the part that decides whether any of it is real. Agents will generate code, perform invasive refactors and run experiments in bulk, but a human still has to pick the workload, protect the holdout, notice benchmark gaming and judge whether a measured gain survives production ^[3]. Luu's ripgrep change pays for compilation in another thread before switching to native code, which suits searches lasting seconds or minutes and is counterproductive for short queries ^[9]. His workload came from his own query history, with a separate representative set used to check that the optimization generalized ^[10]. The 7% travels with that workload and not much further, which makes it a template rather than a patch.

The scale version of the same problem is Michael Malis's pgrust, a from-scratch PostgreSQL rewrite in Rust started with Jason Seibel in early April 2026 ^[13]. Malis reported roughly 250,000 lines in two weeks ^[14], which is about 17,900 lines a day ^[22], eventually running as many as 17 agents in parallel on small test-passing slices to hold merge conflicts down ^[15]. The repository now claims all 46,066 queries in PostgreSQL's default regression suite ^[16] while the maintainers tell you not to store important data in it and say it still has many bugs ^[18]. A regression suite answers compatibility. It does not answer durability, and no volume of agent output changes which question the suite is asking.

So the constraint moves to holdout construction, and holdout sets are made of things teams may not have: real query logs, a representative sample nobody has trained against, and someone senior enough to say a 7% number does not matter here. Luu's own credibility on the cost side comes from having made this calculation the expensive way, in CPU microcode, verification and search indexing, including BitFunnel, a Bing index built for fast constant text ingestion mainly to cut operating costs, whose published throughput varied by workload ^[12]. Workload-specific optimization becoming affordable is the good news. It also means every cheap win now arrives with a private benchmark that only its author can audit.