Running every AppWorld task five times drops a ReAct agent from 77% to 53%

Independent per-run failures would have looked worse than this. A 77% per-run rate compounded across five runs is 0.77^5, about 27% ^[14]. The reported all-five figure is 53% ^[3]. Failures are therefore correlated across runs of the same task.

The mixture is worth writing out. If 53% of tasks pass all five runs, those tasks pass every run by definition, and the remaining 47% have to supply the rest of the 77% average: (0.77 - 0.53) / 0.47 = 0.51 ^[15]. Tasks that miss consistency still pass about half their runs, near 2.6 out of 5. The write-up does not split that 47% into tasks that never pass and tasks that flip, and the two groups want different fixes.

The dev.to post argues temperature is the wrong suspect. Set it to zero and provider-side variation remains, from batching, hardware, and model updates shipped behind a stable name, and the agent loop then amplifies whatever difference survives: a different tool call at step three changes the observation at step four, and by step eight the trajectory is somewhere else ^[10].

So the remedy targets a step. The analyzer needs five recorded trajectories per task to diff before it can locate the divergence ^[4], and the guideline it produces is narrow: "when the search returns multiple contacts, filter on the email domain before picking one" is the post's example of the genre ^[6]. Sixteen points on top of 53 is 69, still 8 points below the 77% per-run rate measured before any guidelines existed ^[18]. For that gain to transfer, a workload needs the shape AppWorld has, with tasks similar enough that a guideline stored on one is retrieved on the next. The post's author finds the 13-point unseen-task result more convincing, on the grounds that guidelines can be overfitted to a fixed task list ^[8].

"Pick your N from your volume, not from the paper," the post's author wrote ^[12]. The numbers support him. On a case that flips 2% of the time, a five-run harness sees at least one failure with probability 1 - 0.98^5, about 10% ^[16]. For a 90% chance of catching that case, you need roughly 114 runs of it ^[17]. His example pipeline runs 400 times a day, where a 2% flip rate is eight wrong vendor assignments ^[12].

Five runs per case already multiplies model and tool calls per eval pass by five ^[19]. One hundred and fourteen does not fit in CI. What does fit is the third field in his harness, a Counter over tool-call sequences, which records that a case passed twice by two different routes ^[13].