Five retrieved guidelines lift AppWorld's hard-task success by 14.2 points

Two numbers in the dev.to writeup describe the same hard-task result: a 14.2-point delta on AppWorld ^[2] and a 74 percent relative increase in success on hard tasks ^[3]. Divide one by the other to recover the baseline. 14.2 / 0.74 is about 19.2, so the unaided agent solved roughly 19 percent of hard tasks and the guideline-assisted one lands near 33 percent ^[4]. About two thirds of the hard partition still fails ^[5].

The context claim is narrower than the phrasing suggests. The evaluated agent got the task instruction plus the top five retrieved guidelines generated on a prior run ^[6]. Five rules is a fixed per-call cost, and it does not grow as the agent accumulates experience, because a background consolidate-and-score job merges duplicates, prunes weak rules and boosts proven strategies instead of appending to the prompt ^[7]. Capture sits in an observability layer, Langfuse or another OpenTelemetry-based tool, with pluggable extractors persisting candidate entities ^[8].

For the 14.2 to mean anything on another workload, the environment has to look like AppWorld's: tasks calling about 9.5 APIs across 1.8 apps, with hard cases marked by more complex control flow ^[9]. The guidelines were mined from train and dev runs and tested on an unseen test-normal partition of the same benchmark ^[6]. That tests generalization across tasks on a single fixed API surface. The headline metric is Scenario Goal Completion, which requires success across scenario variants, and the post says SGC gains exceeded raw pass-rate improvements ^[10].

The Consistency Analyzer is the cheaper half of the system to try. It resamples each decision point in one recorded trace with a single call requesting k completions, k=5 by default, needs no ground truth, and never re-runs the task end to end ^[15]. Measuring Pass^5 directly means five complete runs. If decision points track API calls, a 9.5-step AppWorld task costs about ten resampling calls against about 48 agent steps across five runs ^[20].

Whether that diagnosis applies to your stack rests on one claim in the Hugging Face post: greedy decoding and a fixed seed govern how a distribution gets turned into a token, and neither one changes the distribution, so near-ties still reorder under GPU floating-point non-associativity and request batching ^[18]. Pass^k is the pessimistic mirror of the familiar Pass@k, requiring every attempt to succeed, and it sits at or below Mean@k ^[17]. Standard agent evaluation reports Mean@k, often with k=3, sometimes k=1 ^[23].

The dev.to post does not say which model backed the ReAct agent behind the 14.2-point delta ^[21]; the consistency post names GPT-4.1 for its baseline ^[13]. It also cites an unnamed recent MIT study for the claim that 95 percent of pilots fail because agents do not adapt and learn on the job ^[12].

Trying the lite version costs two commands in Claude Code, after which the plugin extracts entities from trajectories, stores them as files on the filesystem, and retrieves them through Claude Code hooks ^[11].