Grading one agent session on five dimensions reloads the same trace five times

A scorer is three parts, according to Warp: a judging prompt, a set of output classifications that define what counts as a pass, and a sampling rate that sets what percent of runs get graded ^[10]. The definition also names the judge model ^[6]. The grade you get back is a classification against a rubric somebody on your team wrote, and the rubric is the part most likely to be wrong.

There is not much to it. A scoring agent loads a prior trace, brings all of that trace's inputs and outputs into context, and prompts a model to judge the run; the output is a classification ^[7]. Inputs there means prompts, tool calls, MCP results and input images. Outputs means the artifacts the agent produced: PRs, specs, screenshots ^[5].

That bounds what a score can tell you. The judge reads the session. It does not read your incident channel. Two of the DORA metrics Warp lists, defect rate and time to fix errors ^[1], describe what happened after the merge. A scorer can catch an agent that padded a task with redundant tests, which is the exact check Warp wrote for its own factory after seeing that failure mode internally ^[9].

Now the cost. Each scorer covers a single dimension ^[6], and each scorer run loads the whole trace ^[7]. Grade one session against the five dimensions the post lists and you read the same inputs and outputs five times ^[17]. Warp says scoring currently accounts for about 3% of total token costs in its internal factory, which it calls a reasonable amount for visibility into agent performance ^[12]. The post does not give the sampling rate behind that 3% ^[18]. For the figure to transfer, you would need traces of comparable size, a judge model priced similarly against the model doing the coding, and a similar number of dimensions per sampled run. At full sampling on five dimensions, the floor is five extra passes over every run you do.

Warp's stated destination is a feedback loop. "Scoring forms the basis of agentic self-improvement, where observer agents automatically suggest changes to improve agent ROI based on past scores of how the factory is performing," the post says ^[15]. Grading is the input to that; whether the suggested changes raise ROI is asserted in the post and not measured. The nearer use is comparison: Warp suggests pointing the same scorers at different models on your own setup ^[16]. Run that way, the rubric measures the models against your own repository.

On adoption, the post is straight about the work. Warp stores the traces and makes them API accessible, potentially in a company's own storage, and says that depending on your factory approach you may have to do some infrastructure work to set this up ^[13]. Rolling your own means a cron-based cloud agent to score prior runs, plus somewhere to store and view the aggregate scores ^[14].