Build1 publisher3 min readPublished
One line of curl is enough to make a site's dashboard log you as ClaudeBot
A site logged twenty requests for its unreachable /.env file from a client naming itself ClaudeBot. GreyNoise counted 824 addresses scanning under forged AI crawler names, and saw no request for robots.txt in the whole campaign.
The Engineer · Build desk

What happened
- In the seven days to September 18, a client identifying itself as ClaudeBot asked one site for its /.env file twenty times, a path that returns 404 there.
- The site's dashboard put ClaudeBot at 68% verified for that week, but the column scores a crawler in aggregate and cannot say which paths the unverified requests hit.
- The real ClaudeBot fetches robots.txt more often than any other path, at 12% of its traffic.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint An allow-list keyed on the user-agent string cannot separate the crawler you invited from a scanner that typed the same header, so the only enforceable version of that rule matches the source address against a published range or reverse-DNS scheme.
- capability The missing robots.txt fetch gives operators a detection rule they can run against logs they already keep, with no vendor feed and no new agent.
- contradiction The 68% verification figure and the twenty .env requests sit in the same dashboard and cannot be joined, so the spoofing case here rests on GreyNoise's 824 addresses and not on the site's own percentage.
- exposure Any site that exempts self-declared AI crawlers from the scrutiny it applies to everything else has handed that exemption to whoever types the name.
A user agent is a header the client writes about itself. It is a claim [1]. The author of the dev.to post had already shown that on his own site two weeks earlier: he requested a page while calling himself ClaudeBot, and his dashboard recorded the visit as ClaudeBot, which took one line of curl [13]. Any firewall rule or plugin keyed on that header is trusting whatever the client typed.
The dashboard does carry a verification check, and for the seven days ending September 18 it put ClaudeBot at 68% verified [4]. About a third of ClaudeBot-labelled requests that week could not be confirmed as Anthropic's [5]. The number cannot be joined to the twenty /.env requests, because the column aggregates per crawler and not per path [6]. "Verification is per crawler, not per path," the author wrote, and of the question he set out to answer: "I am reporting the limit rather than writing around it" [7]. He says it is the same gap he described two weeks ago and has not fixed [19].
So it is GreyNoise that supplies the evidence that these names are being worn by scanners. On August 31 the firm published counts from July 28 to August 23: 824 addresses running scans under forged AI crawler names, six names belonging to four AI companies, with two Amazon crawler names appearing in greater volume still [8][9]. The paths they asked for were /.env, /.env.production, /.env.bak and /.aws/credentials [10]. Across that entire campaign GreyNoise counted no requests for robots.txt [11]. Real ClaudeBot fetches robots.txt more than it fetches anything else, at 12% of its traffic [12].
That absence is a cheap detection rule, and it transfers only under conditions worth naming. Your access log needs the user agent, the path and the source address on the same line, retained long enough to judge a client over days. You have to be looking at crawler requests at all; the author's point is that filing "AI crawler" away as a separate, benign category is exactly what the campaign is built to exploit [16]. And the signature holds only while scanners do not add a robots.txt fetch of their own, which would cost them one request per host.
The check that does not depend on the client's honesty is the address. OpenAI, Anthropic, Perplexity, Common Crawl and Google all now publish IP ranges or reverse-DNS schemes, and a request that claims one of those vendors and does not match the scheme is worth nothing [14][15]. That is more work than an allow-list of names, and an allow-list of names is what most site owners were told to build after two years of advice to let GPTBot and ClaudeBot and PerplexityBot through [20].
The author ranks all of it below one thing: never serve secrets over HTTP, because when .env is unreachable the scan burns itself on a 404 [17]. On this site /.env returns 404, as does /read-document, the other path in the panel that does not exist [3].
The author calls it a tiny sample and says it is consistent with the campaign GreyNoise measured at 824 addresses [18].
What to watch
- Whether the dashboard's verification check moves from per-crawler to per-path, which would let the author answer whether those twenty requests were Anthropic's.
- Whether scanners start adding a robots.txt fetch to their sequence, which would retire the cheapest signature in the GreyNoise data.
- Whether firewall and CMS plugins that ship AI-crawler allow-lists switch their default from user-agent matching to reverse-DNS or published IP ranges.