Six calls, eleven evaluators, and the pass condition that still lets dead air through

Six scripted calls will not give you a tail. The advice is to report P50, P95 and P99 for each stage of the pipeline ^[15], but at six samples both upper percentiles resolve to the same slowest call ^[2]. So the figure meant to tell you the agent feels unreliable is a single observation of a single script. The median is defensible at that sample size, since it is describing what a normal call feels like ^[16]. The tail is decoration until the call volume is real.

The timing repair is smaller than it looks. Separating model time from synthesis time took two lines of `time.monotonic()` after the first responsiveness number silently included the ElevenLabs call ^[18]. That resolves two of the five stages named in the guidance, and leaves three folded together: time to first audio, speech-to-text and tool latency ^[3][15]. An awkward pause is still attributable to any of them ^[17], and the model alone averaged 3.72 seconds per reply across the six calls ^[19], which is the part of the budget that is now measured rather than guessed.

There is a reason suites drift toward execution. Code evaluators are fast, free and return the same answer every time ^[10], and the behaviours worth asserting that way are enumerable: tool call order, argument values, required disclosures, status codes, excessive tool usage ^[8]. Experience has none of that. It is where most of the build time went, and it is the part transcript-only evaluation does not see at all ^[14]. The judged half degrades quietly too: "Was this a good response" is not an evaluator, whereas a rubric that fails the call if date, time and timezone are not confirmed before the booking tool fires is one ^[11].

That is the actual work in the three-way split ^[1]. Not eleven evaluators, but knowing which of the three questions each one answers, and which question nothing in the suite answers at all.