Handing a coding agent the frozen machine fixed 17 of 33 broken hackathon builds

Exit 0 is the check, and the harness runs it. When the agent stops working, the harness reruns the exact command that failed ^[5]. One machine got to exit 0 by suppressing the error with `@ts-expect-error`. The build passed and nothing was repaired, so the post counts that row separately and flags it for a human before anyone calls it a fix ^[17].

Two further checks exist for the same reason. Once the command passes, the harness measures what changed in the repository and reads the agent's note ^[6]. It then inspects the machine's home directory, to catch a repair that moved the toolchain instead of the code ^[11].

The harness enforces its three rules instead of asking for them ^[9]. Network access is restricted per machine by the platform to package registries and the model API, and the post's line for that is "a prompt cannot argue with a firewall" ^[10]. The agent also runs on the machine itself. The box already has the shell, the toolchain and the broken repo, so the harness only asks a question and checks the answer ^[7]. Fly, which sells these machines, pitches them as "computers for agents" ^[8].

Counting source lines only, excluding lockfiles and regenerated artifacts, the median repair was 4 lines, 11 of the 17 came in under 10, and the largest was 47 ^[14]. The agent's note on one machine said "all 3 problems were version and environment mismatches, not logic bugs", and the author writes that sentence held across most of the 17 ^[16]. Behind it sits the drift the earlier run found from outside: contract code generated with one compiler and a runtime pinned from another, or code written against an SDK version whose exports had since moved ^[15].

The totals do not quite close. 17 and 15 sum to 32, one short of the 33 machines ^[1], and the post separately describes one build that hangs for ten minutes whatever is done to it ^[13]. The failure count moved too: the previous post reported 39 failures, six of them the author's own harness, which is about one in six ^[12][2].

The first 7 machines ran on the biggest model available, the money ran out on a Sunday afternoon, and the remaining 26 ran on the smallest; the author calls that a confound and keeps it in ^[19]. That is honest, and it is thin. He then reran the original 7 on the small model ^[20]. On the 4 of those with a real bug, the large model fixed 4 of 4 and the small model 1 of 4, and its single pass edited a generated type definition that the next regeneration would overwrite ^[21]. For that spread to transfer, a failure population has to look like this one: version and environment mismatch in small projects, measured four machines deep.

About $7 of model time across 33 machines is roughly 21 cents a machine, and about 41 cents for each of the 17 builds that came back ^[3][4]. The project total was just under $16, from $20 loaded with $4.02 left ^[18]. Subtract the run itself and close to $9 went on attempts the author had to repeat ^[5].