Build1 publisher2 min readPublished
Eight stacked repairs drop ProgramDistill's partial-reconstruction success to 32%
ProgramDistill turns 26 working web applications into thousands of verifiable tasks where the specification is the running software itself. Difficulty comes from stacking repairs that depend on one another, and the reported scores track it.
The Engineer · Build desk
What happened
- ProgramDistill specifies a feature by handing a coding agent a fully working reference web application to interact with, then asks it to implement that behavior in an incomplete copy of the app.
- The paper's mine-craft-patch pipeline mined 1,975 replay-verified behaviors from 26 applications and built 4,063 tasks with no human intervention.
- Across nine frontier coding agents, the paper reports GPT-6 Astra at 49.2% and Claude Opus 5 at 28.8% success on cumulative workflows in full-application reconstruction.
- In partial-application reconstruction, reported success falls from 100% to 64.0% and from 96% to 32% as restoration depth rises from 1 to 8.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- capability Task supply for this kind of evaluation no longer depends on how many issues humans will write up, so difficulty can be tuned by dialing restoration depth.
- exposure A buyer comparing agents on issue-style scores now has a second axis to ask about, because more than half of cumulative workflows go unfinished even for the best-reported agent here.
- contradiction The paper's framing is behavior-as-specification, but the experiment it reports varies depth, so a reader looking for proof that written issues flatter agents will not get it from these curves.
- precedent The authors name curriculum-based training as a future use of the same pipeline, so the depth axis is likely to appear as a training schedule before it appears as a leaderboard.
The grader is replay equality. A task passes only when the recorded interaction executes identically after the submitted patch is applied [5]. Every mined behavior comes with a gold patch that makes that replay pass [6]. So the running reference is the specification and the oracle at the same time, and a near miss scores nothing: the paper's own example is dragging a card, where an agent has to infer both the visible effect and the application state that persists afterward [12].
Depth comes from composition. Behaviors carry prerequisites, since a user has to log in before creating an object and create that object before modifying it later [7], and the pipeline chains those lineages into cumulative repairs along a controllable restoration-depth axis [11]. Treat each added step as independent and the reported curves imply per-step success of about 94% and 86%: 0.64^(1/7) = 0.938, and (0.32/0.96)^(1/7) = 0.855 [15]. Independence is the generous assumption there, because one wrong early patch breaks every replay that depends on it.
The full-application and partial-application figures come from settings the paper reports separately [3][4], so they are not comparable.
Prior work in this shape, ProgramBench, had agents probe compiled programs in C/C++, Go or Rust and rebuild them from observable execution behavior, treating each program as a single whole-program reconstruction target [8]. ProgramDistill factorizes below that line. Feature-level mining yields about 2.1 tasks per verified behavior and roughly 156 tasks per application [13][19].
Four conditions have to hold before those percentages say anything about your codebase: a working reference an agent can drive, a replay harness deterministic enough that "executes identically" is a decidable check, features with genuine prerequisite ordering, and acceptance defined as behavioral equality. Interactive web apps supply the ordering cheaply, because login and object creation impose it [7].
The depth-1 endpoints are the part I would not skip past. Both curves start essentially solved [4]. Accumulation is the failure mode: inferring one feature from a running application and patching it into an incomplete one is the part these agents handle. The paper does not report the same tasks respecified as written issues, so what the curves measure is restoration depth [17].
The authors write that they are working on a public release of the benchmark [9].
What to watch
- Whether the public release ships the 4,063 tasks with a runnable replay harness, and whether the 26 applications are named.
- Whether anyone reruns the same task set with written issue specifications as a control condition.
- Whether the full paper breaks failures out per depth increment or only reports end-to-end success.