ChatGPT named a vendor in 28.6% of its opening search sub-queries across 3,842 runs

ChatGPT handles a search prompt by writing sub-queries, called fan-out queries, and retrieving web pages for them ^[1]. The opening sub-query is written before any result has come back ^[1]. A vendor name in that query did not come from retrieval. The dev.to write-up's interpretation is that ChatGPT draws on brand-category associations it already holds while it forms its search plan ^[10]. Independent B2B SaaS analyses describe what those plans tend to contain: searches for an official brand page, for documentation, for pages restricted to one domain, and comparisons across several brands ^[8].

Radyant published the dataset openly and put an interval on its headline number, so the figure can be checked ^[2][4]. The 28.6% sits in a reported band of 25.6% to 31.6%, or plus or minus 3.0 points ^[3][1]. If all 3,842 runs were independent draws, a standard 95% binomial interval would be about plus or minus 1.4 points ^[2]. Treat the 615 questions as the sample and it widens to about plus or minus 3.6 ^[5]. If the reported band is also a 95% interval, it is much closer to the question-level figure ^[6]. With about 6.2 runs per question ^[3], I'd expect repeated runs of one question to open in similar ways. The band's width fits that, and it puts the effective sample nearer 615 questions than 3,842 runs ^[6].

For the figure to describe another category, that category's buying questions would need to resemble Radyant's 615. They would also need to reach the model as it behaved during the August 11 to 17 collection ^[2], in the same language and from the same exit location. The language and location conditions come from the arXiv study, which tested 234 logged-out ChatGPT and API runs across languages and countries ^[11]. The dev.to summary does not say which languages or exit locations Radyant's runs used.

Radyant's own numbers limit how far the opening-query finding reaches. In the other roughly 71.4% of runs, the opening query named no vendor ^[5]. Brands also reached answers that no sub-query had named ^[7]. The write-up says a company missing from the initial framing can still be found through retrieval or named in the answer, though it may have fewer routes in ^[13]. Retrieved pages still count; they work alongside the model's choice of what to look for ^[14]. In about 1,099 of the 3,842 runs, that choice included a vendor before retrieval began ^[4].

Given the instability the arXiv team measured across identical runs ^[17], a screenshot of one favorable chat proves only that one chat happened. The write-up suggests testing realistic buying, comparison and alternative-to prompts, repeating them across runs while recording answers and any visible fan-out queries, and running them in the languages and locations where the business sells ^[15]. It also suggests checking that the company's site and public materials tie the brand to the category, use cases and terms its customers use ^[16].