Topic
An evaluation approach that checks each AI agent control layer—prompt, context, harness, loop, and graph—rather than judging only final output quality.
One developer's 'could not tell' exit code stopped a false green on a hold, then exposed four new ways a check can mislead. Each looks like a clean result, so agent evals that adopt the third value need controls that force every outcome.
Reality
Five agents built to cheat a ledger-scored refund benchmark found three scorer holes, including a $50 cap that scored $120 in split refunds as $0.00. An outage, too-good results and a fact-check found the other three.
Reality
Huntress ran seven Rails tasks past Claude Fable 5.1 and watched it copy hand-rolled code out of its own repository. Changing the harness, in three cheap steps, took API recall from 48 percent to 100.
Reality
A dev.to post credits Anthropic with running the same model and the same prompt under two harnesses, 20 minutes and $9 for a broken result against six hours and $200 for a working one. The hourly spend barely moved.
Reality
AWS has added three skill-focused evaluators to Strands Evals and Bedrock AgentCore Evaluations. Two of them call a model once per invoked skill, and the third is a deterministic check that exists only in Strands.
Reality
NVIDIA's developer blog sets out the five-level rollup from step to benchmark and the two scores that read the same trace. Step-level says where the chain broke. End-to-end reads the environment and says whether the refund posted.
Reality
LangChain clocked TypeSafe AI's Jev at 0.44 seconds and $0.00035 per call against three LLM judges on the same eval set. Whether that price transfers depends on how much structure your traces already have.
Reality
A dev.to FAQ proposes failing any agent session whose diff touches a test path without the ticket ID in a signed allowlist. Both scripts in the post are labeled unexecuted, so the false-positive count is yours to find.
Reality
COGEXT's author pushed 150 samples from cookbooks, DEV posts and Hacker News through his own extractor. The unbiased 120 yielded one promise, and the 13 in the enriched set averaged 0.79 confidence with a single deadline between them.
Reality
License Referee resolves each npm dependency to an SPDX id and looks up a ruling for the exact dependency-into-project direction. On its 22-question test set, all three configurations agreed on the verdict and one invented its source.
Reality
ProgramDistill turns 26 working web applications into thousands of verifiable tasks where the specification is the running software itself. Difficulty comes from stacking repairs that depend on one another, and the reported scores track it.
Reality
A dev.to post lays out a coding-agent eval protocol that hashes the token and tool-call envelope alongside the tasks, so editing a cap changes the run id. No executed runs are published with it.
Reality
Turing Post talked through agent feedback with Salesforce's chief AI scientist at Dreamforce. A correction lands in one of four places: the context, persistent memory, the software around the model, or the weights.
Reality
A developer ran three change tasks in one Python and TypeScript product against two code-index tools, and counted a run only after the focused test passed and then failed again with the defect put back on purpose.
Reality
Coding-agent pass rates get re-quoted after every prompt edit. A dev.to protocol freezes a seeded holdout split before any tuning, and at its minimum size the published number gets too coarse to show a 13-point gain.
Reality
AWS's system prompt optimizer hands a reflector agent a shell tool and a directory of scored production traces, and offline batch evaluation plus a live-traffic A/B test decide which of its proposed edits reach the agent.
Reality
A Microsoft developer blog post documents a Dev Proxy knowledge evaluation that blocked web tools and curl, then passed questions about recent versions. The agent had been reading a local source checkout.
Reality
The harness swaps a large tool response for a file path and a ten-line preview, truncates stale write arguments at 85 percent of the window, and summarises only when there is nothing left to move to disk.
Reality
Toolmetry edited only the strings that tell an agent what each MCP tool does and how to call it. On a single published run, its three test servers finished between 96.4 and 100 percent for $4 of API spend.
Reality
The durable output of OpenAI's new agent cookbook is an eval suite generated from human and model feedback on five runs of one fictional company, plus a handoff file that tells Codex what to change next.
Reality
Build · September 15, 2026 · 1 publisher
Build · September 13, 2026 · 1 publisher
Build · September 10, 2026 · 1 publisher
Build · September 10, 2026 · 1 publisher
Build · September 10, 2026 · 1 publisher
Build · September 9, 2026 · 1 publisher
Leadership · August 31, 2026 · 1 publisher
Build · August 26, 2026 · 2 publishers
Invest · August 23, 2026 · 1 publisher
Build · August 23, 2026 · 1 publisher
Build · August 23, 2026 · 1 publisher
Build · August 22, 2026 · 1 publisher
Build · August 16, 2026 · 1 publisher