The classifier that put gotchas in 13.6% of AGENTS.md files catches barely a third of them

Divide 13.6 by the recall the same study reports for that category and the floor moves. On the 55-file held-out English set, the pattern rules caught between 32% and 38% of the gotcha content a human reader found, because they miss casual watch-out prose ^[7]. Take both numbers at face value and the share of files carrying something gotcha-shaped lands near 36% to 43% ^[1]. That correction assumes misses across the corpus scale like the held-out set, and that false positives on that one rule are few. The post reports precision and recall for the classifier as a whole, 92% and 70% on English files, and notes that the per-category numbers carry that error ^[5][16].

One measurement skips the classifier entirely. The author pulled 347 entries from sections headed Gotchas, Common Pitfalls or Troubleshooting in an earlier 507-file snapshot, then hand-labeled 120 of them. Of those, 58% were readable straight from the repo, 34% were generic advice such as remembering to install dependencies, and 8% needed experience or the outside world ^[10]. Eight percent of 120 is about ten entries ^[2].

The kind of sentence at issue is small and specific: the README names one port and production uses another, a mismatch the study found in a corpus file that otherwise looks complete ^[20]. The explanation offered for why those sentences go unwritten is about incentives. "The fix is visible in the code; the sentence is a liability nobody wants to maintain," the author wrote ^[11]. One of the more thorough files in the corpus runs 34 KB over 27 sections with 19 separate boundary markers, and still contains nothing you could only learn by running the thing ^[12].

The generation experiment transfers most directly, if your file is model-written. Across 11 repos and three prompt styles, prompts that listed the topics explicitly produced all nine categories. Prompts that left the slots implicit skipped workflow in 11 files out of 11. Naming workflow by name got it written 3 times out of 3, with real content ^[13]. That works for every slot whose content sits in the repository. For the 8%, it fails, because a model reading your code has no access to last week's upstream outage ^[19].

The labeling method sets the limit on all of these numbers. Labels come from pattern rules over headings and body text, and the blind labeling was done by a single model-driven annotator. Repos were found through AI and agent topics plus Chinese keyword search, so the Chinese set came out 97% Chinese by construction ^[16]. The English bias of the rules shows up in one figure: agent_meta recall on Chinese files is 26%. Those files put agent rules in the second person, as in "you are the dispatcher, not the executor", and the body patterns are all English ^[8]. Rules and human reader agree on all nine categories for 12% of files, in both languages ^[9].

The rule set is readable and recomputable from the published dataset, which is the stated reason for keeping a language model out of the labeling loop ^[1]. When compare told the author that one of the nine categories was missing from a file he had written himself, the category was there. The heading had used the tool's own slot name instead of natural language. He documented the miss in the repo instead of patching it quietly ^[17].