Skip to content

Build1 publisher3 min readPublished

An availability word from the clerk now revokes Oathra's earlier booking confirmation

Oathra scores an agent phone call with separate code that reads the callee's turns, and its maintainer spent this week fixing the two natural English confirmation shapes that the checker itself got wrong.

The Engineer · Build desk

Illustration accompanying An availability word from the clerk now revokes Oathra's earlier booking confirmation

What happened

  • Oathra, an open-source runtime for AI agents that make phone calls, decides whether a call succeeded with a separate piece of code that reads the other party's words, and the model that made the call gets no vote.
  • Its maintainer lays out five clerk replies to the same request to book a table for two, scores only one of them as a booking, and says a model asked whether the call succeeded would say yes to at least four.
  • Until the day before the post, a clerk who confirmed the table and then said "Sorry, that day is fully booked after all" was reported by the runtime as a completed booking.
  • Every release runs 10,000 seeded adversarial dialogues against a hard gate of zero false completions.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • constraint Counting filled fields cannot separate a booking from an agreement to call back, so a completion check has to track proposal, acceptance and revocation across turns and scope negation to the clause.
  • cost Test coverage is per-language work: the daily-use Japanese path was clean while two English shapes were wrong, so each language a fleet answers calls in needs its own adversarial set and its own maintenance.
  • contradiction A green zero-false-completion gate sat alongside a live false completion, so the figure bounds the shapes someone thought to seed and not the shapes a restaurant clerk will produce.
  • exposure Revocation fires on a short vocabulary of availability words, so a callee who kills the booking in any other phrasing still leaves a confirmed result on the record for whoever shows up at 7:30.

The interesting case is where the negation lives. In "We can hold September 25 at 7:30 pm for now, but it is not confirmed yet", the clause carrying the time is the clause being negated [7]. Extract per utterance and the time lands in one variable while "not confirmed yet" lands in another, and the negation is lost on the way [8]. So the extractor has to work at clause level.

The third reply is worse, because the agent finishes the trap itself. The clerk says "The only thing left that evening is 9 pm. Would that do?" A number was said, and it was offered as a question [9]. The agent's next turn is usually "9 pm works, thank you", so if the extractor took 21:00 from the clerk's turn you get a fully populated booking that nobody agreed to [10]. Oathra holds the clerk's turn as a proposal until the caller accepts it and the clerk acknowledges the acceptance [11].

Two of the five shapes were mislabelled by the checker itself until this week, and they failed in opposite directions [26]. The fully-booked reversal failed open, because the old retraction pattern required the refusal to name the booking, in the manner of "we cannot take the reservation" [13]. The plain confirmation failed closed: "You are all set for September 25 at 7:30 pm, party of two" returned nothing at all, not "unconfirmed" but empty [18]. The engine kept two lists, one for the callee agreeing to a value the caller proposed and one for the callee confirming the booking, and "you are all set" was in the second but not the first [19]. A confirmation is bound to the values settled when it was spoken, so a confirmation with no settled values behind it is stale and gets dropped [20].

That bug surfaced by accident. "I only found it because I pasted an English log into my own public checker while writing a comment on someone else's thread," the maintainer wrote [21]. Three of the five natural English confirmations worked; the Japanese side, used daily, was fine [22]. "The lesson is not about regexes; it is that the language you don't test in is the language that's broken," the maintainer wrote [23].

The seeded corpus behind the release gate is the author's own: the five shapes plus voicemail, transfers, hold-then-reply, dialect confirmations and wrong restatements [25]. A confirmation followed by a bare availability refusal cannot have been in that set while the runtime was still scoring it as completed [28]. For the zero to say anything about your callees, your callees' phrasings have to be in the set.

PR #37 is narrow on purpose. An availability word from the callee (full, private hire, closed, "fully booked") revokes a confirmation spoken strictly earlier [14]. "We can't take cards" after a booking stays a payment remark, and the ordinary "7 pm is full but 7:30 is free" before a booking is untouched [15]. The tests run in both directions [16]. It is still a vocabulary, so a clerk who kills the booking in words outside it leaves a confirmed result standing.

The case for taking the verdict away from the model rests on those five replies as the maintainer reads them; the post does not report a measured comparison between a model's verdicts and the checker's [27]. I would take the separate check anyway, on the strength of the first reply: a check that counts filled fields passes it, and the date, the time and the party size are all correct, while the clerk has only promised to call back [5][6].

What to watch

  • Whether the 10,000-run gate is reported per language, given the English shapes broke while the Japanese path passed.
  • Whether the availability vocabulary grows past full, private hire, closed and "fully booked" as more clerk phrasings arrive.
  • Whether anyone publishes a model's verdicts alongside the checker's on the same five replies.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories