The MCP test that matters: a log tool that fetched the data and then said it failed

A benchmark published on dev.to pointed one agent at the hosted Appwrite and Vercel MCP servers and gave both the same job: deploy identical applications, inspect the builds, diagnose a deliberate failure, ship a second version, and recover from it ^[1]. The headline number is not the score; it is a bug the author says Appwrite shipped into the run, where the runtime log tool fetched its data successfully and then told the agent it had failed ^[2].

That is the worst class of defect in an agent tool surface, and it is invisible to any catalogue-based evaluation. A missing tool produces an honest dead end. A tool that returns success and reports failure teaches the agent that the observability path is broken, so the agent stops asking, and the operator inherits a diagnosis built on a false negative. By contrast, the author reports Vercel's dedicated log tools filter properly and that it reported runtime failures better ^[3].

The setup was tighter than most vendor comparisons. Both servers ran over OAuth from the same Darwin 25.5 arm64 client, the first call to each tool was discarded so nobody paid for a cold start, the two servers were alternated so neither benefited from going second, and writes landed in a throwaway project ^[4]. The workload was a static site with identical index.html and marker.json files deployed from the same source archive, plus a Next.js 15.2.8 SSR app with deterministic build markers and controlled 200, 400, and 500 responses ^[5]. Six stages ran in the same order on both platforms: discover the workspace, deploy both apps, inspect logs and analytics, recover from an intentional build failure, publish v2, attempt a rollback, and handle 50 requests at concurrency 10 ^[6][7]. Correctness and completeness carried 30 percent of the score, with compactness, discoverability and call economy, latency, and safety and lifecycle control at 15 percent each and error recovery at 10 percent ^[8], which puts twice the weight on correctness as on any other single category ^[9]. Appwrite won five of the eight scored tasks, and the author is explicit that the three it lost, it lost badly ^[11].

Tool count told you almost nothing here. Vercel exposes 33 direct tools spanning projects, deployments, logs, analytics, agent runs, collaboration, domains, purchases, and deployment protection ^[12]. Appwrite exposes 992 operations across 81 services behind four meta-tools for context, search, execution, and docs ^[13], roughly thirty times the catalogue ^[14], at the usual price of one extra routing call before an unfamiliar operation ^[15].

Lifecycle is where the gap became operational. Appwrite deployed v2 to the existing Site with a build cache hit, switched the active deployment, and rolled back to v1 in 1.32 seconds, with the health endpoint returning v1 immediately ^[16]. Vercel's inline deployment tool created a new project for v2 and therefore reported no previous cache, and its tested MCP surface exposed no rollback, no redeploy to an existing project, no project deletion, and no general environment-variable management ^[17]. The consequence the author names: both Vercel projects from the run are still live, because nothing in the 33 tools can delete them ^[18]. Appwrite also required confirm_write=true before every mutation and refused the first unconfirmed create ^[19].

Watch whether Appwrite's log tool bug gets fixed, and treat the aggregate scores with care: the author says the two combined totals were close enough that the gap tells you nothing, and the per-category numbers behind them were never published ^[10].