Topic
Instability of model verdicts at temp=0 with a fixed seed, run-to-run benchmark variance, and variance added by inference routers.
Debashish Ghosal got HivePlane's agent certification to repeat across three runs by swapping in deterministic agents and a mocked judge. The stable verdict covers the control plane's plumbing, and whether an LLM-driven agent answers correctly is now outside the certificate.
Reality
The model's answers wobble by a few hundredths on identical input, so the checkable properties live in a TLA+ spec and a Rust quorum rule that counts only votes whose margin clears a measured noise floor.
Reality
IBM Research calls the 24-point drop between passing once and passing five times the consistency gap. Independence would have predicted a 27% five-run rate, so the failures are clustering on particular tasks.
Reality
An LLM can answer the same prompt two different ways, so the acceptance criteria a product manager used to hand off no longer survive the handoff. Product Talk's guide argues the person who defines good output now has to measure it.
Reality
A vendor-sponsored walkthrough proposes a behavioral gauntlet in place of a demo. The method survives the sponsorship better than the pass thresholds shipped with it.
Reality
Salesforce's engineering team argues production agent evaluation has to verify system state, not replies, because the same code and prompt can either call the refund tool or merely say it did.
Reality
A verify-on-read experiment rerun across 14 live models on a fingerprinted 50-fact set found false-accept rates up to 0.38, and run-to-run noise wide enough to swallow a prompt fix.
Reality