JavaScript navigation is now a discoverability decision, and 41 days of crawl logs show the cost

SEO engineer Vinicius Stanula ran a 41-day crawl experiment on a site of roughly 1,000 pages and logged 30,180 requests from 27 distinct crawlers, documenting the setup and results in a field experiment posted on LinkedIn ^[1][2][3][5]. Pages reachable only through JavaScript-injected links were not discovered by GPTBot, ClaudeBot, PerplexityBot or any other AI crawler observed in the test, while Googlebot did get through ^[6][7].

The design was blunt enough to be useful. Of 21 top-level categories, 11 used standard HTML anchors visible in the page source and 10 relied on links injected by JavaScript that were absent from the initial HTML response ^[4]. That puts roughly 48 percent of the top-level taxonomy behind client-side navigation ^[2], and if pages were spread evenly across categories, on the order of 480 URLs ^[3]. Traffic was not the limiting factor: 30,180 requests over 41 days averages about 736 crawler hits per day ^[1].

Googlebot's result is the one that gets misread. Googlebot and GoogleOther did reach the JavaScript-linked content, but their traversal was not equivalent to the HTML-linked side of the site, and coverage weakened at deeper levels of the hierarchy ^[7][8]. The AI-oriented bots did not find the hierarchy pages available only through JavaScript at all ^[9]. So "Google renders JavaScript" was never the same guarantee as "Google reaches everything," and it was never a guarantee about anyone else.

The switch is the part that closes the argument. Once all links became plain HTML, GPTBot found 250 new pages in the first 48 hours, Bingbot added 89, and Google added no new pages, consistent with its earlier ability to reach the JavaScript-linked areas ^[10][11][12]. GPTBot's haul was about 2.8 times Bingbot's over the same window ^[4]. Nothing about the content changed. Only the route did.

Stanula is careful about what this does not prove: it says nothing about whether any individual AI system will use, cite, or train on a given page ^[13]. What it shows is narrower and still consequential. Content can be publicly viewable in a browser and simultaneously unavailable for crawler discovery through the site's own internal architecture ^[14][15]. Crawlers generally start with the HTTP response and read its HTML; a link added only after a browser executes a script leaves a non-rendering bot with no route to follow ^[16].

The remediation is unglamorous. Use standard HTML anchors for essential category, product, documentation and editorial pathways ^[21]. Inspect the delivered HTML rather than the browser-rendered DOM before shipping ^[17]. Test crawl paths at depth, because a bot reaching a top-level page does not mean it reaches the tail ^[18]. Server-rendered HTML or static generation can keep the JavaScript experience while exposing anchors in the initial response ^[20]. None of this requires abandoning a framework; it requires a crawlable baseline underneath one.

The governance point is worth separating from the engineering one. Organisations may have legitimate reasons to restrict automated access, but accidental exclusion is not the same thing as deliberate control ^[22]. Sites with large estates carry the most exposure, because documentation hubs, knowledge bases, ecommerce category trees and enterprise resource centres depend on hierarchy pages to expose large numbers of URLs ^[23].

Watch your server logs after the next front-end release, since bot behaviour can shift with a technical change and logs are where that shows up first ^[19]. This is one site, one operator, one 41-day window ^[1][2], so the number to treat as provisional is the 250-page, 48-hour spike ^[10]; the pattern to treat as load-bearing is that a route absent from the raw HTML was, for these bots, no route at all ^[6].