Build1 publisherNot yet confirmed elsewhere3 min readPublished Updated
Pennyforge probe finds 40 of 186 registry-listed MCP servers complete an anonymous tool call
Pennyforge Studio found that 40 of 186 servers in one MCP registry slice completed an anonymous tool call. Most failures were credential gates or schemas that understate required inputs, so a registry listing says little about whether an agent's call will succeed.
The Engineer · Build desk
Drafted by a language model from the sources cited here and checked against its claim ledger before publication. How we use AISend a correction
What happened
- In the registry's a-b slice, 87 of the 186 listed servers returned HTTP 401 to the opening initialize call and the rest errored or timed out, leaving 78 that answered.
- Each answering server got an initialize, a tools/list and one tools/call with empty arguments to a read-style tool declaring no required inputs, under 12-second caps.
- Thirteen servers let an anonymous client initialize and list tools, then demanded a token, key, OAuth sign-in, personal link or account at tools/call.
- Six servers declared a tool with no required properties, then rejected the empty call with errors such as "property_coverage requires address OR both latitude and longitude".
- Three months after the stateless 2026-07-28 protocol revision shipped, only 9% of answering servers had migrated to it.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint A liveness check that stops at initialize or tools/list would report all 13 call-gated servers as working, because their credential demand appears only when a tool is actually called.
- exposure Agent frameworks that build arguments from inputSchema.required will break on about one in seven answering servers, and the missing argument is named only in free-text error messages.
- decision For a team holding credentials, the anonymous pass rate may understate what will work: 102 of the 186 listed servers asked for credentials somewhere, so the probe has to be rerun with real keys.
The client was anonymous throughout, identified only by one declared User-Agent [4]. Against that bar the tally was 40 healthy, 19 call errors, 15 list-only, two tools/list errors and two initialize errors [14]. Forty is 51.3% of the servers that answered and 21.5% of the 186 listed [23][24]. Pennyforge's prose says "A little under half of the servers that answer at all let an anonymous client complete a real tool call" [26]. Its own count comes out slightly over half [23].
The gates vary. Six servers wanted a bearer token, two an API key, one an OAuth sign-in, one a personal link, and three an account signup, all three run by one operator [7]. Two more returned bare 401s that do not say which credential they want [7]. One "free" probe key costs $19 for 30 days, a price the post found in an error message [15]. Another server runs on its operator's expired API key and rejects every caller, its author included [16].
Beyond the six schema mismatches, a client-side pass found five more endpoints whose failures had the same shape [18]. That makes 11, or 14.1% of the answering cohort [25]. The probe picks its tool by trusting the declared schema, so these servers land in the error column. Pennyforge wrote that an agent framework that trusts inputSchema.required fails systematically against these servers [19].
Two entries break before the protocol starts. One ships an unexpanded URL template, and every client gets a Cloudflare 530 "Origin DNS error" [8]. Another answers initialize at /mcp with its HTML marketing page and an HTTP 200 [9]. A health check keyed on status codes would pass it.
Speed is not the problem. Median initialize was 453 ms, median tool call 468 ms, and p90 1.8 s [20].
The 51% describes one workload: an anonymous client against one alphabetical slice of one registry [1][4]. For it to transfer, the a-b names would have to resemble the rest of the registry, and the agent would have to connect without keys. The post covers only the a-b slice [1]. A credentialed client meets a different population. Add the 87 initialize-stage 401s to the 15 servers gated later, and 102 of the 186, or 54.8%, asked for credentials somewhere [2][6][22]. Only 21 errored or timed out outright [21].
Sessions show a similar gap between what servers declare and what they do. 69 of 78 servers never issue an Mcp-Session-Id header, though the revisions they declare still describe sessions [11]. The spec makes session IDs optional with a MAY [11]. The first probe requested the 2025-06-18 version and so under-reported newer ones. The 2026-10-05 probe asked for 2026-07-28 to get each server's real maximum [12].
This is careful work. The run took about 20 minutes and cost $0 [3], and the post ships its repro and raw data with a second dated point on 2026-10-06 [13]. I'd copy it before wiring a server into an agent, and I'd keep the raw error text, since the 13 call-stage gates and all 11 schema mismatches showed up only at tools/call [5][25].
What to watch
- A Pennyforge run beyond the a-b slice would show whether the 51% anonymous pass rate holds across the whole registry.
- Whether registries start making a real tools/call before listing, or pull entries such as the unexpanded URL template that fails for every user.
- The 2026-07-28 migration share at Pennyforge's next dated re-probe, against 9% now.