A hand audit of 45 register questions found 34 hedges and three confabulations

Eleven of the 45 ungrounded calls asserted something about a named company. Three of those were confidently wrong, which is more than one answer in four ^[1]. The quotable ratio, 3 in 45, measures how often the model spoke at all: 34 of the 45 calls were hedges ^[2].

The report will not bank that as a product win, on the grounds that beating a model which already said "I don't know" is not beating anything ^[7]. Per case, the model hedged on all three trials for 9 of the 15, 60% of the set ^[6]. On those nine there is nothing for a tool to improve.

The audit itself is the part worth copying. The classifier's overcount of wrong answers ran better than two to one against the human read ^[3], and the named cause is that it cannot separate an assertion about a specific company from the same words used generically ^[9]. A judge that reads "status (active, dissolved, etc.)" as a claim about a particular company has a taste for drama. Every one of the reclassified calls was a failure mode the project had already logged once ^[8].

On the with-tools side, the strict rule matters more than the headline. Three raw passes over the eligible cases came in at 19, 23 and 20 of 27 ^[16]. Requiring all three trials to agree drops the result to 18 of 27, or 67%, against a per-trial mean of 77% ^[15][16], a ten-point haircut for instability alone ^[6]. The report calls the best-to-worst gap eighteen percentage points; 23/27 minus 19/27 is 4/27, which is 14.8 ^[4]. The argument for the strict rule survives the correction.

The VAT case is the mechanism to take away. Asked whether a UK company is VAT-registered, the model called search_company to confirm identity, then answered that Companies House does not publish VAT registration data ^[17]. That is true of the search response, which carries no VAT field, and it is not what the record says, where vat_registered is null ^[17]. Answering from the tool needs a second lookup and the willingness to report a null. It passed 1 of 3 ^[17]. A tool call in the transcript is not evidence that the field was read.

Coverage is the other half of the case. Only 15 of the 31 authored cases can be asked without tools, because a case about which of three registers to pick is meaningless when there are none ^[4], leaving 16 cases with no ungrounded baseline to beat ^[5]. On the VAT case both arms did share, the toolless answer was "Yes, Tesco PLC is VAT-registered ... since its taxable turnover far exceeds the threshold", which the report describes as confident, reasoned and unsourced ^[18]. Its summary of what a register actually buys is provenance rather than accuracy ^[19].

The 34 hedges transfer only under conditions the report states. The model is claude-sonnet-5, measured on one day ^[2]. Fifteen hand-authored prompts across three national registers are the whole question set ^[1][4]. Nothing in the context asked for a decisive answer, because the system block was omitted from the API call along with the tools ^[4]. Three trials smooth sampling noise without removing it, as the report notes ^[10]. And when the model did answer under a disclaimer, the shape was the headcount case: register access disclaimed, then "330,000-360,000 employees" offered as general knowledge on two of three trials ^[12], which is precisely the caveat an agent pipeline strips off ^[13].