Build1 publisher3 min readPublished
DeepDeck's WebMCP ablation checks the OFF arm's tool list is empty before counting tokens
The DeepDeck benchmark runs one task twice, once with the WebMCP bundle installed and once with the tool list forced empty, freezing the model configuration in between and leaving correctness unscored unless you supply an answer you checked yourself.
The Engineer · Build desk
What happened
- The author of DeepDeck, an MIT-licensed macOS desktop project built on DeepSeek Harness, added a benchmark that asks whether exposing a structured browser tool reduces the time and tokens a task needs.
- Both arms keep the ordinary browser tools, the custom WebMCP bundle is installed only in the ON arm, and the OFF arm disables WebMCP and then checks that its tool list is empty.
- Each run writes comparison.md and report.json carrying both final answers, total tokens with raw cache accounting, agent elapsed time separate from application startup, and tool calls, errors and actual WebMCP usage.
- The URL-based experiment needs no Docker, registered benchmark site or task YAML file, while the bundled 49-task corpus still requires its own site infrastructure.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost Measuring the benefit costs inference: at --n 3 the runner completes the task six times through your own provider account, and the bill falls on whoever wants the answer.
- decision A team picking WebMCP on token counts is not picking on price. The totals fold in cache reads and writes, and the post notes a provider can consume more tokens and still cost less.
- constraint Clean profiles drop signed-in cookies and the runner cannot reset a remote backend, so authenticated and state-mutating workflows sit outside what this harness can measure.
- contradiction The published batch ran one repetition per arm in fixed ON-then-OFF order, which is weaker than the alternating-order design the current runner enforces, so the batch cannot settle the question the runner was built for.
When the `ablate` subcommand starts, it takes an isolated snapshot of the configured model so the two arms cannot drift, creates fresh browser profiles and sessions, and alternates which arm runs first across repetitions [3]. With `--n 3` that is six agent runs of the same task, three on and three off [6]. Each one makes model calls through your configured provider [5]. If the runner finds no tools at all, it writes a not-applicable report and exits before spending anything on inference [7].
The framing the post starts from is that successful tool registration only proves availability: it does not tell you whether the agent used the tool or whether the answer was right [2]. The sample tool reads the page's `h1` and returns the URL, and it does not contain the expected answer [21]. A tool that contained the answer would turn the ON arm into an oracle.
Correctness is explicitly unscored unless you pass `--expected-answer`, and the post says a completed run is not automatically a correct run [10]. That check ignores case, normalizes whitespace, and tests text containment; the post is clear that it is not a semantic evaluator [11]. Containment passes any answer that includes the string, so it fits a heading or a stock number and not a judgement. The expected string stays with the runner and is never sent to the agent [11].
Getting to a first run means `pnpm install`, `pnpm build:desktop`, and a working model configured in DeepDeck before the runner snapshots it [22]. Pass counts on the author's published batch were 49 of 49 in both arms for Terra, 48 ON against 47 OFF for DeepSeek, and 49 ON against 48 OFF for Hy3 [14]. Summed, the ON arms passed 146 of 147 task runs and the OFF arms 144 [16]. Restricting the efficiency comparison to tasks all three models passed in both arms drops three of the 49 [15]. The post reports those pass counts and does not publish the token or elapsed-time figures for the 46 shared tasks [23].
The author calls the batch exploratory: one run per task per arm, a fixed ON-then-OFF order, and provider-default reasoning with the effective level unrecorded [17]. For a published delta to transfer to your own site, you would need the same provider and tokenizer, comparable reasoning effort, and pages that behave the same on a clean profile. The post says providers, tokenizers and environments differ, and that its results page documents environment and scorer limitations [18].
On what he wants to test next, the author wrote: "The insight I want to investigate further is that fast individual inference does not guarantee fast task completion." [20]
What to watch
- Whether the author publishes token and wall-clock deltas for the 46 shared-pass tasks, with repetitions and the reasoning level recorded.
- Whether sites that already ship native WebMCP tools get measured with the custom bundle omitted, which is the configuration site owners care about.
- Whether the expected-answer check gains a semantic scorer, since containment limits which tasks can be scored at all.