Loading today’s stories
Build1 distinct publisher3 min readPublished
A demo proves an agent worked once under conditions you chose. The New Stack's remedy is a small fixed scenario suite with pinned fixtures, and the pinning is where most of the actual work sits.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
Follow any of these and your For You feed starts watching them — no settings page required.
build
CSPM ends up as the intake queue for agent-provisioned infrastructure1 distinct publisher
build
Testing a skill means running the scenario again on the next model version1 distinct publisher
build
A refactoring benchmark stops the best agent at 41.2%, and the tests are the story1 distinct publisher
build
Amodei's compute ceiling: owning the weights is not sovereignty if the GPUs are rented2 distinct publishers
Start with the layer you freeze, because that choice decides whether the gate does anything. The guidance says to freeze the documents and the tool responses used during a run, and to pin account state to a known snapshot, with the policy version and the agent's permissions treated as equally part of the fixture [16]. Documents and retrieval output are not the same layer. If a scenario replays what the retriever returned, the code that assembles context never executes, and the retrieval configuration change that opens the story sails through [2] [7]. If the scenario replays the corpus and lets the retriever run against it, the same change surfaces as a different cited source. Freeze one word differently and you get two very different harnesses.
The second adoption cost is trace access. The useful distinction in the piece is between result and process: an agent can give the right answer after retrieving the wrong document, finish a task after calling a tool it did not need, or search outside the customer's scope, and the transcript still reads as a success [11]. Asserting on that requires the runtime to emit retrieved document identifiers, tool call names with arguments, and the permission decision at each call. If your framework hands back messages and nothing else, those assertions cannot be written, whatever the evaluation tool claims to score.
The arithmetic matters here too. The New Stack recommends ten real tasks over a large benchmark of prompts your users never send, and a first set small enough for one person to maintain [13]. Each scenario pins five things [16]. Ten scenarios is fifty pinned artifacts [19]. Production failures then join the suite as permanent regression cases [18], so the count only rises, and every policy revision invalidates a slice of it. The ten-task ceiling looks like modesty, but it is a maintenance budget expressed as a scenario count.
The multi-turn design is the part worth copying verbatim. Ask for an account change, supply the missing identifier in the next message, confirm in a third, and assert that the agent carried the identifier and the proposed change forward without dragging unrelated details into the final action [15]. That is an assertion about state, not about phrasing, which is why it should survive a model swap. The same instinct shows in the advice to put exact assertions on high-risk rules while allowing the surrounding wording to vary [12]. Brittle string matching on prose is how suites get deleted.
What the source does not supply is measurement. There is no rate at which model upgrades break citation behaviour, no named tool, no cost per run. The evidence is one well-described failure pattern plus a release criterion: if a team cannot reproduce a run or a material regression in a high-risk workflow, the product is not ready to pass the gate [5]. Narrow enough to hold.
Ranked by verification strength, evidence, and original report placement.
The New Stack describes a common pattern: a team builds an agent, gives it a few representative questions in a test chat, tries a slightly harder prompt, records a demo, approves the change and ships it.
A demonstration tells you an agent worked once under the conditions you happened to give it, and does not fully establish whether the next version will consistently preserve the behaviour users and operators need.
"If it can't reproduce a run or a material regression in a high-risk workflow, the product isn't ready to pass the release gate."
A repeatable evaluation system runs fixed scenarios along the product's execution path and records sufficient evidence to determine whether a release should proceed.
The evaluation should exercise the code that assembles context, the tools the agent can call, and the permissions the runtime enforces.
Before choosing an evaluation tool, the team should write down the jobs the agent performs, the limits around each job, and the outcomes that fall outside the product's accepted operating boundaries.
Distinct publishers with included, body-backed reporting in this cluster.
1 article · September 4, 2026
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
One voice, well-specified, unverified
Every detail traces to a single New Stack piece, and the strongest material in it is craft judgment rather than observation: the five things to pin, the ten-task ceiling, the deterministic permission checks. The failure that justifies the whole argument — a citation dropped and a wrong customer lookup tool chosen after a retrieval change and a model upgrade — arrives without a team, a date or an incident report behind it. Specific advice, no independent confirmation that any of it has been run.
Nobody named
Not one team, product, evaluation harness or suite appears anywhere in this reporting — not even anonymously. There is no count of adopters, no regression the method caught, and no tool it was run with, so there is nothing to measure uptake against.
Prescription slightly ahead of proof
Modest, not egregious. The piece is unusually careful about what a demo does and does not show, which pulls the gap toward zero. What pushes it positive is the confidence of the remedy: 'the product isn't ready to pass the release gate' is stated as settled practice, while the maintenance bill — roughly fifty frozen artifacts before you have covered ten tasks — goes unmentioned, and no example shows the suite catching anything.
No stake visible
We cannot see who benefits if readers follow this. There is no vendor, no sponsorship marker, no author affiliation and no product in the text — the one place a commercial interest would show, the choice of an evaluation tool, is deliberately left blank. Guessing at motive from a publisher's beat would be invention.
Coherent, thinly sourced
We are confident about what is being recommended and much less confident that it has been tried at scale. The internal logic holds up — the fixture list follows from the reproducibility requirement, and the arithmetic on a starter suite follows from the fixture list. But a single publisher, an illustrative opening failure and zero named practitioners cap how far this can be trusted as a description of how agent teams actually ship.