Invest1 publisher2 min readPublished
Amazon blocks the two OpenAI crawlers behind 50 million daily shopping queries
New disallow rules for ChatGPT-User and OAI-SearchBot went into Amazon's robots.txt without an announcement. Amazon earns roughly $56 billion a year from advertising that bills only while shoppers are on its own pages.
The Investor · Invest desk

What happened
- Amazon's robots.txt now disallows ChatGPT-User, the agent that fetches live web information for answers, and OAI-SearchBot, the crawler behind OpenAI's SearchGPT search engine.
- Independent e-commerce analyst Juozas Kaziukenas spotted the new disallow rules and posted a screenshot on LinkedIn, and Modern Retail confirmed them by reviewing the code behind Amazon's site.
- A working paper from OpenAI's Economic Research team says those agents help power around 50 million shopping-related queries a day, including requests such as a laptop recommendation under $1,000.
- On Amazon's latest earnings call, CEO Andy Jassy said the company is having conversations with third-party shopping agents and expects it will find ways to partner over time.
- Amazon sent Perplexity AI a cease-and-desist letter over its Comet browser, which lets users ask an agent to find and buy items on Amazon; Perplexity called it an aggressive legal threat.
Compiled by The InvestorSomething wrong?How this is made
Why it matters
- cost Amazon is declining the ChatGPT referral traffic that Walmart, Target and Etsy are collecting, and paying for that choice in lost sessions to protect an ad business that only bills while a shopper is on Amazon.com.
- constraint OpenAI has to answer around 50 million shopping questions a day without live access to Amazon product pages, so its commerce answers rest on the retailers that signed deals.
- contradiction Jassy's talk of finding ways to partner and the new disallow lines point in opposite directions, and with Amazon declining to comment the robots.txt file is the only statement of intent on the record.
- precedent Amazon's position that agents should respect service provider decisions turns robots.txt into the boundary, and the Perplexity letter shows the charge it will bring when an agent crosses it: computer fraud.
Amazon's advertising business runs at roughly $56 billion a year, about $153 million a day, and it depends on people browsing Amazon.com instead of shopping through ChatGPT, where ads can be bypassed [6][18]. Set against that, the 50 million shopping-related queries a day that OpenAI's own economists attribute to those agents come to 18.25 billion a year, so the entire Amazon ad business is worth about $3.07 for every ChatGPT shopping question [7][19]. The $3.07 is a ratio between two published figures, not an estimate of what an Amazon listing inside a ChatGPT answer would earn.
The reading that Amazon is fencing its catalog off from ChatGPT commerce has to survive an awkward fact: the file was already unfriendly. GPTBot, the crawler that scrapes content for model training, was blocked before these lines went in [4]. Amazon has spent recent months shutting out third-party crawlers tied to Meta, Google and Perplexity as well [5].
Jassy's complaint about outside agents is accuracy. Many of them, he said, give wrong pricing, delivery and inventory information: "We have to find a way, though, that makes the customer experience good," he said [12]. Amazon's public position on the file is that third-party shopping agents should "operate openly and respect service provider decisions" on whether to participate [16]. Amazon and OpenAI did not respond to requests for comment [17].
I would expect the lines to stay. Amazon's $56 billion of advertising depends on the browsing session, and the ChatGPT referral traffic is going to Walmart, Target and Etsy [6][8], while Amazon spends on its own AI shopping tools instead of opening the door [20]. Scot Wingo, who writes the Substack Retailgentic and founded ReFiBuy, said Amazon "doesn't want to just be the back end of the internet" and that "They want to be the front door, and they're going to fight this for quite a while, would be my guess" [13]. Two developments would break that view: the disallow lines coming back out of the file, or Amazon naming the third-party agent partner Jassy described. The other possibility is duller, which is that these blocks are aimed at price scraping and automated checkout generally, with OpenAI the latest name on a list that already ran to four companies [5]. Tyler Cowen asked Sam Altman at a Berkeley conference last month whether Amazon would fold and join or fight back and do its own thing. "I have no idea," Altman said, and added, "If I were them, I would fight back" [10].
What to watch
- Whether Perplexity complies with Amazon's cease-and-desist or takes the Comet purchasing dispute to court.
- Whether other large retailers add the same disallow rules for ChatGPT-User and OAI-SearchBot to their own robots.txt files.
- Whether OpenAI signs further in-chat checkout deals of the kind it has with Walmart.