Build1 publisher2 min readPublished
A default Cloudflare rule answered an MCP client with an HTML block page for three days
Debugging an agent endpoint has to start on the wire, because a client that reports a parse error is describing the last thing that touched the bytes rather than the machine that swapped them out.
The Engineer · Build desk
What happened
- The MCP endpoint answered a client speaking JSON-RPC with HTTP/2 403, content-type text/html and server: cloudflare, which is a Cloudflare interstitial rather than anything a JSON-RPC client can decode.
- The connector surfaced the failure as a parse error, so for three days the team checked its own JSON-RPC, content types, status codes and protocol revision, all of which were correct.
- A hundred-line mimic server with the same response shape, no auth and no database, exposed through an ngrok tunnel, listed tools on the first try for the same connector.
- Holding body and endpoint constant and varying only User-Agent, ClaudeBot and GPTBot drew 403 and 4,543 bytes of HTML while Claude-User, Claude-SearchBot, anthropic-ai and curl/8.7.1 drew 200 and 156 bytes of JSON.
- The block came from a Cloudflare managed rule aimed at AI crawlers that ships on by default, which nobody on the team had enabled and nobody had disabled.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost The debugging bill falls entirely on the application team, because the rule admits every tool they would reach for to prove the endpoint healthy, so each successful check sends them back into their own code.
- constraint With Claude-User and ClaudeBot sharing the Anthropic AI Crawler bucket, a site cannot write "refuse training, serve agents" as a category rule and must maintain an allowlist of individual agent names instead.
- exposure Any endpoint sitting behind this default fails in a shape no client can report honestly, since cf-mitigated is absent and the only machine-readable trace is server: cloudflare plus a cf-ray id.
- decision Bringing up an MCP server behind a CDN now warrants one request sent with the agent's own User-Agent before anyone opens the serialization code, because that is the cheapest test that distinguishes the two failures.
A JSON-RPC client handed 4,543 bytes of HTML fails inside its own deserializer, and the deserializer is the one component in that path that lives in your repository [9][1]. The status line and the content-type are right there on the wire, but the string a developer actually reads names the parse, so the search starts in the code and stays there [2]. The block page, for its part, asked a program to enable cookies [1].
The loop broke when the team built something that could not contain their bug: a hundred lines, `application/json` on POST, `405` on GET, no auth, no database, tunnelled through ngrok [6]. One result cleared three suspects at once, the connector, the protocol and their own response shape, and left only the segment between connector and application that is not in the repository [7]. The author puts the mimic server at an hour of work against the rest of the week [8]. The adjacent fix shipped by itself, a GET that returned JSON where the Streamable HTTP spec calls for `405` when a server offers no SSE stream on that verb [4]. Keeping it unbundled is the reason the next result meant anything [5].
The refusal weighs 4,543 bytes against 156 for the real answer, about 29 times the size [18]. If you log response bytes per route, that ratio is visible without reading a body at all. One link in the chain is not established by the write-up: which User-Agent the failing connector actually sent [19]. Its own table shows `Claude-User/1.0` returning 200, so the rows that reproduce a 403 are `ClaudeBot` and `GPTBot` [9]. The attribution to the managed rule rests on the author's account of finding it on by default [10].
The category problem outlives the incident. Anthropic runs three crawler identities on purpose, `ClaudeBot` for content that may contribute to training, `Claude-User` when a person has just asked a question, `Claude-SearchBot` for search indexing, with three robots.txt entries so a site owner can refuse training and stay reachable [12]. Cloudflare's AI Crawl Control, in the Security section, files `Claude-User` and `ClaudeBot` together as Anthropic AI Crawler while giving `Claude-SearchBot` its own Anthropic AI Search bucket [13]. Anchor Browser, an agentic browser driven by a person, is filed as an AI Crawler as well [14]. So the finer categories exist and the one that matters for agent traffic is the one that collapses [15]. The configuration the team ended up with, refusing training crawlers and passing user-initiated traffic, is a policy somebody had to author by name; the default authored a different one for them [16].
What to watch
- Whether Cloudflare splits Claude-User out of the Anthropic AI Crawler category the way it already separates Claude-SearchBot into AI Search.
- Whether managed-rule blocks start emitting cf-mitigated or any machine-readable refusal signal an agent client could log.
- Whether MCP clients begin reporting HTTP status and content-type alongside deserialization errors instead of only the parse failure.