Empty Kaggle proxy captures scored DeepSeek-R1 at 17% on a benchmark where it reaches 100%

The benchmark's author wrote that the first thing it caught was the harness, not a model ^[19]. Kaggle's Model Proxy finished the runs, but for many of them it captured no response ^[4]. An empty string cannot be read as OK or PROBLEM, and the grader counted each one as a wrong answer ^[5]. The author wrote that "a benchmark that does not audit its own capture is measuring the plumbing, not the model" ^[12].

The scorer now flags each empty response and drops it ^[7]. Dropping is a sound first fix, but it changes the denominator. GLM-5 and Gemini 3.5 Flash Lite are scored only on the items the proxy captured ^[15]. Their percentages compare cleanly with the others only if the proxy's drops have nothing to do with the items, such as snippet length or how long a model takes to answer. For the models the proxy dropped, the author sent the identical prompt through an OpenAI-compatible gateway and scored it with the same grader ^[15]. The cleaner fix is to retry until text comes back and then grade the full set. For DeepSeek-R1 that retry was worth 83 points ^[3].

The set is small. With 28 items ^[1], one verdict is worth about 3.6 points ^[1]. The top models are separated by one or two snippets, as the single over-flagged items for Claude Sonnet 5 and Gemini 3.8 Flash show ^[8]. Grok 4.20 has a bigger gap: 100% with reasoning on and 68% with it off ^[10]. That is about nine items ^[2].

The most useful result in the post is where those nine items came from. According to the author, they clustered in deliberately paired items. A parameter-free transform before the split is fine, and a fitted scaler before the split is a leak. A constant fill for a missing value is fine, and a column-mean fill is a leak ^[11]. With reasoning off, the author wrote, "It matches the position of a line instead of what the line computes." ^[13]

The small model gets it wrong in the other direction. The author wrote that Gemma 4 31B "reads the confident claim, agrees, then walks past the leak" ^[14]. For any of these percentages to transfer to a review bot on your own repository, your inputs would need to look like the benchmark's: a short snippet, a confident claim about it, a forced binary verdict, and a flaw from the leakage family ^[1][2][3]. Most pull requests do not arrive with a caption saying the method is sound. I'd expect the direction of error to transfer more readily than the percentages. At the frontier, that error is a PROBLEM verdict on correct code ^[8].

The design holds up where it matters. Because the set is balanced 15/13, a model that answers PROBLEM to everything lands near 50% ^[3]. Grades are on the verdict, but the written reason is kept as well. According to the author, on the top models that reason names the right failure almost every time ^[16]. Every item, verdict and the grader are public on Kaggle ^[17]. The author disclosed that Claude helped build the benchmark and draft the post. The capture bug turned up because the author read the raw model outputs by hand ^[18].