Loading today’s stories
Build2 distinct publishers3 min readPublished
The replay only works on runs Warp's infrastructure already recorded, and correctness is graded by a judge model against a rubric you write. Warp's own 30-task bake-off cost $2,130.57. It finished in under four hours.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
Replay is the load-bearing mechanism here, and it works only because Warp Factories writes down more than the diff. The infrastructure stores every agent trace and its metadata: the prompt, the full conversation, the Git state before and after the run, the resulting PR, and any other generated artifacts [4]. The factory itself is a file, factory.yaml plus supporting agent definitions, which is what makes an A/B between two configurations reproducible rather than anecdotal [5]. Rebuild the starting state, vary one field, run it again [3]. The output is a recommended overall configuration, a task-by-task breakdown and tradeoff plots that can feed routing rules, sending one class of work to a cheaper model and reserving the expensive one where it earns the difference [21].
That is also the product's fence line. You can benchmark the tickets you already ran through Warp Factories, and no others [4]. Enterprises can keep the traces inside their own security boundary [6], which answers the residency question and leaves the dependency intact.
Grading is where I would spend the scepticism. A Scorer is an LLM-as-a-judge loop that assigns a grade against a rubric the user defines [7], across correctness, code quality, efficiency, verbosity and cost [8]. So "correctness" here is a judge model's read of a replayed run, graded against the rubric you wrote. Where your acceptance criteria are expressible as tests, you hold better ground truth than the scorer does, and the rubric should say so. Where they look like adherence to a Figma mockup [8], a judge is the only thing that scales.
Now the arithmetic. Thirty tasks across five models is 150 runs, so $2,130.57 is about $14 each at one repetition per task [11]. Cheaper than the meeting where the same question gets settled from vibes. The 3 hours 46 minutes is 226 minutes, roughly 1.5 minutes per run in aggregate, which only happens if the matrix runs wide in parallel [12]. Repetitions are a launch-time knob, and Warp notes more of them cut variance and add cost [23]: three reps on that same matrix comes to about $6,392 [14]. Against that, Warp reports lowering internal cost per PR by about 75 percent [13], so one $2,130 run clears break-even once the re-routed work was previously costing around $2,841 [15].
Two numbers in circulation do not sit still. RuntimeWire, reading the WarpBench case study, reports Warp swapped its auto (genius) routing, which sent complex tasks most often to Opus 5, for Grok 4.6 High as the primary implementation agent [16]. The launch post, published a day later [17], names GPT 5.6 Sol (high) as the winning model on the same internal benchmark [18]. Both can hold if one is scoped to implementation and the other to the overall pick, but the gap is a fair proxy for how long any answer stays current.
Read the harness line twice as well. The launch post opens by saying Benchmarks work across frontier and open-weight models and, soon, across harnesses such as Warp, Claude Code and Codex, while the walkthrough further down says you can vary the harness to test Claude Code against Codex [19]. WarpBench was run in Warp Agent [10].
For the 75 percent to transfer, your ticket mix has to resemble that S-to-XL spread over Go, React and Rust [9], your acceptance criteria have to be judgeable by rubric, and your agent spend has to make a four-figure experiment a rounding error. Lloyd's argument that public task sets may be contaminated by training data is his assessment, and Warp has not published an independent analysis of the benchmarks it references [20]. The weaker claim, that private codebases differ from each other, needs no analysis and is enough to sell the layer [25].
Ranked by verification strength, evidence, and original report placement.
Zach Lloyd launched Factory Benchmarks inside Warp's agent infrastructure on September 3rd, giving engineering teams a way to compare coding models against their own historical tasks and route future work based on the results.
Warp describes Benchmarks as launching in Warp Factories early access, the first model bench generated from a team's own coding tasks, similar in idea to public benchmarks like SWEBench and Terminal Bench but run on the team's own tasks and context.
Teams can create a benchmark by selecting previous agent runs stored in Warp Factories or asking an agent, through the Warp Factories MCP, to assemble a representative task set; Warp replays each task from its original code and Git state while holding the relevant factory configuration constant.
A Scorer is an LLM-as-a-judge evaluation loop that takes an agent run and assigns a grade based on a rubric the user defines; Warp also uses Scorers for its self-improvement loops.
Built-in scorers cover correctness, code quality, efficiency, verbosity and cost, and teams can write rubrics for requirements public coding benchmarks rarely capture, including adherence to a Figma mockup or the quality of end-to-end tests.
The WarpBench case study says Warp replaced its "auto (genius)" routed configuration, which sent complex tasks most frequently to Opus 5, with Grok 4.6 High as the primary implementation agent after the benchmark's findings.
Distinct publishers with included, body-backed reporting in this cluster.
runtimewire.com
1 article · September 3, 2026
warp.dev
1 article · September 3, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
build
Developer habit, priced at $965B: what Anthropic's run actually proves1 distinct publisher
build
Three frontier launches in a day, all pitched on price. Open weights set the ceiling.4 distinct publishers
build
OpenAI's cheap tier becomes a routing problem: Terra $2/$12, Luna $0.20/$1.20, seats untouched2 distinct publishers
science
Frontier scores at $2/$6: Grok 4.6 ties GPT-5.6 on one evaluator's index for a fifth the output price1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
One vendor's documents, one outlet doing the division
Every load number in this story — 30 tasks, $2,130.57, 3 hours 46 minutes, 75%, GPT 5.6 Sol on top — originates with Warp or with Lloyd's case study. RuntimeWire read the same two documents and did useful arithmetic on them, which is checking, not verification. Nobody outside the company has replayed a task, and the correctness scores at the heart of the product were assigned by Warp's own judge model rather than an independent evaluator.
Dogfood, plus a waitlist with credits attached
The only team known to have used this is Warp. That use is real and consequential — it moved production routing twice inside a few weeks — but it is the vendor testing the vendor. There is no named customer, no disclosed run or seat count, and the go-to-market is still an early-access application backed by $10k of free usage.
Firmer numbers than the record can carry
'The first model bench generated from your own coding tasks' and a clean 75% do a lot of work here. Underneath, the same post gives two different 75% claims, the case study's number is 63% (62.5% unrounded), the underlying chart covers four dates rather than a period, and the compliance gain was scored by the seller's own judge. The winning model changed twice in weeks, which undercuts the implied stability of a benchmark-derived routing decision. The gap is one of precision and independence, not invention: the architecture described is specific and checkable.
The scorekeeper is selling the scoreboard
Warp built the benchmark, ran it on itself, graded it with its own judge, published the results, and is recruiting early-access teams with $10k of credit — and the results happen to demonstrate exactly the saving its buyers want. RuntimeWire names this dynamic well for model providers and rival agent vendors, then reports the launch from the same vendor's documents, which is its own mild exposure.
Clear on what shipped, thin on what it delivers
What Warp built is documented in unusual detail — the primitives, the replay path, the routing rules, the launch-time knobs — and two publishers agree on it. What it produces for anyone other than Warp is unknown, and the cost and savings figures come from a single interested party with no external replication. High confidence in the mechanism, low in the outcome.