OpenAI's improvement loop compiles five traced runs into a rerunnable Promptfoo gate

Start with the eval-generation step. Human feedback and model-written critique over the five traces are compiled into a Promptfoo suite that can be rerun later ^[4][5]. The notebook then runs that suite as a validation gate over the current agent behaviour ^[6]. So the gate encodes a reviewer's judgement of five runs against one fictional company's diligence materials ^[3].

That is a regression net over observed behaviour. It will fail a change that breaks something the agent did well in those five runs. A failure mode the agent never produced is not in the suite, because the suite was distilled from the traces ^[5]. For the coverage to transfer, your worst inputs would have to resemble the notebook's diligence questions over financial exports, customer data, contracts, security notes and board materials ^[2].

The 20-minute number is a claim about OpenAI's configuration. The notebook says to budget about 20 minutes for a full run with the default five traces, and names Step 3, the traced agent calls, and Step 7, the HALO analysis, as usually the longest ^[10]. It also says model latency and network conditions will move that up or down ^[10]. Model choices sit in one cell so you can substitute cheaper ones and trade quality for cost ^[12].

Nothing replays from cache. The example is deliberately live-only, and trace generation, model critique, eval generation, validation and optimization all use fresh model outputs ^[9][15]. Validating a change three times therefore costs about an hour of wall clock and three sets of billed tokens ^[16].

The second artifact is the handoff. The notebook defines the harness as the full contract around the model, including instructions, tools, routing, output requirements and validation checks ^[8]. A HALO pass over the traces, the feedback and the eval results ranks the next harness changes and writes codex_handoff.md under ARTIFACT_DIR, carrying the full diagnosis, the ranked recommendations, the evidence behind them and the implementation guidance Codex needs ^[7]. Because the checks are part of the harness, a ranked recommendation can propose editing the gate that produced the evidence ^[17].

Adoption costs two runtimes. The Python side is openai, openai-agents and halo-engine; Promptfoo runs through npx, so the machine needs Node.js with npx on the path ^[11][18]. If the eval suite is the artifact you keep, Node is in your CI whether the team wanted a second runtime or not.

How far the loop runs unattended is left to the developer ^[13]. OpenAI describes a reviewed loop, where the system proposes the change set and a developer approves the diff before merge, as a common starting point, and says the same handoff can support deeper automation as the eval gate becomes more trusted ^[13]. Compared with examples that stop at traces or at evals, this one keeps traces, reviewer judgment, generated evals, optimization and the implementation handoff inside a single runnable loop ^[14].