Ahrefs logged zero requests in May to 97% of the valid llms.txt files it found

The validator bug described in the angeo.dev writeup is the kind anyone building a checker will hit. The spec recommends a short summary in a blockquote under the title ^[21]. The check looked for a line starting with `>` and never read what came after it ^[22]. A store with no meta description emitted a bare `>` and nothing else, and the tool certified the file as valid ^[23]. The fix is to require non-whitespace content after the marker.

One finding in the Ahrefs data bears on whether to publish at all. On domains where the file did not exist, AI bots never requested the path; a crawler that wanted it would ask and collect 404s ^[6]. The file gets read when a client has been configured to fetch it, and not otherwise.

The 3% that were fetched can be quoted in either direction. About 19.5% of fetches came from named AI tools, with GPTBot first and Claude-Code second, ahead of every AI search and assistant bot ^[7]. Roughly 12% came from the industry inspecting itself: GEO and AEO tools, llms.txt checkers, researchers ^[8]. Chrome's Lighthouse llms.txt audit was about one fetch in a thousand ^[9]. Separate reporting of the same dataset puts AI retrieval bots at around 1% of total requests ^[10]. For the 19.5% to transfer to your site, your pages have to be what sits at the top of that list: developer documentation that a coding assistant reads. A shop that wants to be cited in a shopping answer is looking at the 1% figure.

That pattern is close to what Google has said. John Mueller described llms.txt in June 2025 as not being done for search, closer to a stopgap that saves tokens for AI coding tools reading developer documentation ^[11].

Quality is the weaker part of the record. Chris Humphrey fetched /llms.txt for the top 10,000 Majestic Million domains in June 2026; of the 1,050 that returned HTTP 200, 313 were serving an HTML soft 404 ^[13]. Of the 737 real files, about 55% had the recommended shape of a title, a summary, and at least one section of curated links ^[14]. That works out to roughly 405 well-formed files against 1,050 successful responses, about 39% ^[24].

The August revision changes what publishing one involves. Link relations are the main addition: a page can declare where its Markdown version lives, and which llms.txt describes it ^[16]. Both Markdown URL forms are now allowed, `page.html.md` and `page.md` ^[17]. An llms.txt at a subpath covers the pages under that path, and the most specific file wins ^[18]. `## Optional` no longer carries mechanical semantics ^[19]. Links should lead to LLM-friendly content, meaning the Markdown versions of pages where those exist ^[20].

In my view the honest reading is to ship a correct file on a documentation site, where the named fetchers actually are, and to leave the per-page Markdown generation until a client asks for it.