Meta's training crawler outpaced its citation crawler 39 to 1 on a single WordPress site

The three user agents do different jobs, and Meta documents which is which. Its description of meta-externalagent says it "crawls the web for use cases such as training foundation AI models or improving products by indexing content directly" ^[5]. meta-webindexer "navigates the web to improve Meta AI search result quality for users" ^[6]. meta-externalfetcher pulls individual links at a user's request, which the operator takes as evidence that somebody asked something a few seconds earlier ^[7].

The counts add up exactly. 6,027 from the training collector, 155 from the indexer, zero from the fetcher: 6,182, all of Meta's traffic on the site ^[25]. That is 38.9 training requests per indexer request ^[24], or 2.6% in the other direction ^[28], which the operator rounds to about two percent ^[13]. The window logged 11,818 AI crawler requests in total, 8,979 of them on pages with writing on them ^[1][2], so Meta's 6,182 is 52.3% of the traffic ^[23] and beats every other vendor combined by 546 requests ^[26].

Whether that split shows up in someone else's logs depends on who controls each side of it. Meta sets the training collector's schedule. Readers set the fetcher's, because it only runs when a user asks Meta AI for a specific link ^[7]. A small WordPress site can pass a month without that happening, and the operator says as much: "One site is not a census" ^[21]. On the same logs, Perplexity's traffic is 93% user-initiated and OpenAI's is 21% ^[14][15].

The identity problem sits under all of it. Meta publishes no IP ranges and no reverse-DNS scheme for these crawlers, so the operator has no way to confirm that any of the 6,182 requests came from Meta, and the dashboard shows a dash where other vendors get a verification percentage ^[17][16]. The operator does not think the requests are forged, and wrote: "I am reporting a number I am not able to check, and that should be said out loud rather than buried" ^[18].

The referral side of the record is thinner than the crawl side. It comes from an earlier month, when the site was read about ten thousand times and two humans arrived, one from ChatGPT and one from DuckDuckGo ^[19]. The operator argues that the largest contributor to the read count was the vendor structurally least likely to appear in the arrival count ^[20].

He also sets aside whether training collection is theft, calling that a separate argument ^[30], and limits the finding to the ratio his instrument shows ^[22].