Leadership1 publisher3 min readPublished
Prices and payments a customer acts on belong in a calculator the chatbot calls
Omni Calculator says any number a customer acts on should come from a deterministic tool, after its benchmark scored AI models at 48.4% to 70.4% on math. Its own Toronto BMW example went wrong at the input, so a calculation engine protects operators only when the right data reaches it.
The Board Room · Leadership desk
What happened
- An Omni Calculator contributor argued in Entrepreneur that any figure a customer will act on should come from a deterministic tool that the AI model calls.
- Omni's third ORCA benchmark of free-tier models scored ChatGPT 5.3 at 48.4% accuracy on math, Claude Sonnet 4.6 at 53.2% and Grok 4.20 at 70.4%.
- A salesperson called to revoke that offer, and the dealership reinstated it only after CBC News asked for comment.
Compiled by The Board RoomSomething wrong?How this is made
Why it matters
- decision Before building, product teams have to sort every number their assistant emits by whether it must be reproducible, then wire the ones that must to a script or calculation API.
- constraint Checking inputs becomes a separate job with its own owner, because an engine guarantees only the formula and will return a mis-fed figure identically on every run.
- exposure With customer service among the top AI uses at small US firms, the reversal finding bites in the channel where customers routinely dispute a quoted figure.
The 100-run rule is a question the team asks before it builds anything. According to the Omni Calculator contributor, the first step is to decide whether an output must be reproducible [2]. If the same request run 100 times must return the same answer, that answer has to come from a deterministic tool the model calls [2]. Nobody has to run a request 100 times to apply it. The source's list of numbers that must be reproducible covers price quotes, shipping estimates, tax figures, dosages, unit conversions and financing payments [3]. Email drafts and ad copy fall outside it, since they usually have no single correct answer [4].
The routing is simple to describe. The model reads the question and picks a tool. A script or calculation API computes the value, and the model writes the reply around that figure [5]. For a mortgage lender, the visitor's income, debts, down payment and rate go to something that runs the amortization formula the same way every time [6]. The author names Wolfram Alpha as one tool a model can call for the job [7].
The case for keeping models off the math rests on research by the author's own company, and readers should weigh it with that in mind. Omni Calculator sells Omni Calculator Builder, now in public beta, in which a model writes calculator logic that runs on Omni's deterministic math engine [8]. In its ORCA benchmark, even the top-scoring model, Grok 4.20, got 29.6% of answers wrong [1]. The author's explanation comes from how the models work: a model predicts text one token at a time and does not hold a number's value the way a calculator does [15].
An engine also changes what happens on the second ask. In the benchmark, Claude and ChatGPT abandoned correct answers when a user replied "Are you sure?" [11]. A formula that runs the same way every time [6] gives back the same figure the customer just challenged.
The board-deck version is that routing numbers to an engine removes the liability. That version is incomplete. In the source's own Toronto anecdote, the C$27,162.79 that Quinn, a BMW dealership's chatbot, offered for a customer's car was the balance he still owed on his loan, handed to the bot by mistake [13]. Nothing was miscalculated. The bot repeated a number it had been given. An engine fed that wrong field would have returned the same wrong offer on every run and passed the 100-run test. The same limit applies to Omni's product, where the model writes the calculator logic and the engine runs it [8]. Broken formulas were among the recurring faults the benchmark found [10].
I think the rule is sound for the calculation step even after discounting the author's commercial stake. A formula run the same way every time does not need a vendor's benchmark to justify it. The published ORCA figures cover free-tier models [9], and the piece does not report how paid tiers score.
What to watch
- Independent tests of paid-tier models on the same calculation, rounding and formula tasks, since the published ORCA figures cover free tiers only.
- Error data from Omni Calculator Builder once it leaves public beta, particularly cases where the model-written formula itself is wrong.
- Any ruling or regulator action holding a business to a chatbot-quoted price; the Toronto reinstatement followed a press inquiry, not a ruling.