One site's Next.js logs show OpenAI's crawlers running JavaScript from September 25

Next.js App Router pages prefetch the links they contain, and each prefetch URL ends in ?_rsc= plus five random characters ^[7]. That string is not in the served HTML. It appears only when something renders the page and runs its JavaScript ^[7]. A ?_rsc= request from an OpenAI address is therefore direct evidence of execution ^[7]. The operator did not trust the user agent alone, a habit more sites should have, and dropped any request outside gptbot.json, searchbot.json and chatgpt-user.json as a fake ^[5].

The counting method is good engineering. Each prefetch carries the page that triggered it in the Referer header, so the number of distinct Referers is the number of pages rendered ^[10]. Counted that way, rendered pages rose about 10x for search and well over 100x for training ^[10].

The GPTBot numbers point to renders happening away from the origin. Of the 4,309 pages GPTBot rendered on September 28, only 28% had been fetched from the server by anyone that day, and the HTML is not cached at the CDN ^[11]. That leaves roughly 3,100 renders with no same-day fetch behind them ^[1]. The operator's inference is that OpenAI re-renders HTML it stored earlier in its own headless browser, and only the prefetches fired during that render reach the server ^[12]. OpenAI's crawler docs say that when a site allows both bots, results from one crawl may be used for both purposes ^[13].

A Vercel and MERJ study in December 2024 found that none of OpenAI's three bots executed JavaScript ^[3]. These logs contradict that for at least one operator's sites. They show execution, but they do not show whether text that exists only after the browser loads it ends up in ChatGPT answers.

The closest evidence is referral data from @soho, the freelance marketplace that is the operator's main site ^[4]. The number of visitors arriving from ChatGPT did not change. From about September 25 they landed mostly on job pages instead of blog posts, and daily signups rose about 1.7x ^[18]. Over the same stretch, job pages rendered by OAI-SearchBot went from 2 a day to between 36 and 108 ^[18]. "I cannot prove one caused the other," the operator wrote ^[19].

The address lists moved during the same week. The operator's own server returned 403 to some of OpenAI's requests from September 22 to 24 ^[21]. gptbot.json changed on September 22 and chatgpt-user.json on September 25 ^[22]. The operator's advice is to pull allowlists from OpenAI's list URLs instead of keeping them by hand ^[22].