The only check available is address matching. A vendor publishes the addresses its crawlers run from, the server compares the source address on the request against that list, and a match is the proof [4]. Nothing inside the request itself carries it. A user-agent header is a claim the client types about itself, and anyone can type it [21], which is how three curl requests from Japanese home broadband ended up filed under Anthropic, ClaudeBot, training [1][3].
The interesting arithmetic is what is left after the unverifiable traffic comes out. Of 13,491 crawler requests over the thirty days to September 4, 2026, 6,947 came from Meta, ByteDance and Amazon crawlers for which no IP ranges and no reverse-DNS scheme are published, so 51 percent of the traffic had nothing to compare against [5][7][2]. A further 2,226 requests were checkable in principle but not by this instrument, whose bundled range snapshot was taken on July 20: Common Crawl published a list on August 11, and Anthropic published one on August 18, replacing an explicit statement that it did not [11][12][13][14]. Remove both groups and 4,318 requests remain that the plugin could actually adjudicate. Roughly 2,200 of those verified [6][3]. Where the check was both possible and current, close to half of the requests naming a verifiable crawler did not come from that crawler's published addresses.
There's a more portable lesson in the snapshot failure: it reflects a design choice many tools make. The plugin never talks to the network, so vendor ranges ship as a bundled snapshot and go stale with no symptom [11]. Anthropic's list existed for 17 days before the fix landed on September 4, and through those days the dashboard kept reporting no method published [12][15][4]. After the fix, the count of crawlers the instrument can verify went from eleven to fifteen, and the same pass turned up a Google crawler that had been marked verifiable for months while its ranges were never bundled [15][16]. The author writes that he would rather not have found this while writing an article about verification [20].
Freshness checks turn out to be harder than they look: the lists themselves are published badly. Amazon does serve a page of Amazonbot addresses, but the contents only appear after JavaScript runs [9]. ByteDance's obvious candidate URL returns HTTP 200 with a 404 page in the body, so a checker keyed to status codes will record that ByteDance publishes a list [10].
The Perplexity section is where the supplied text gives out. 984 requests called themselves Perplexity-User and 103 called themselves PerplexityBot, checked against ranges Perplexity publishes and the author had in hand [17]. The headline says none could prove it [18]. The body says the share arriving from Perplexity's published addresses was not low [19], and the text breaks off mid-sentence on OpenAI before the table appears [22]. Two readings and no table, so that row stays unresolved.
For these ratios to transfer to your logs, your crawler mix would have to look like this one. meta-externalagent alone is 5,728 of 13,491 requests, 42 percent, and it sits in the group with no verification mechanism at all [8][5]. A site whose bot traffic skews to Google, OpenAI and Anthropic would post a much higher verified share with nothing about the method changing. What does not vary is the direction of the error. A blocklist or a licensing claim keyed to agent strings prices the 10,390 requests that landed on pages with writing on them [5] at whatever those requests said they were.
