Build1 distinct publisher3 min readPublished
A 41-day test switched off sitemaps and breadcrumbs so client-side navigation was the only way in, and the crawlers behind ChatGPT and Claude never arrived. Googlebot did, then needed 41 days to notice the fix.
The Engineer · Build desk

Compiled by The EngineerSomething wrong?How this is made
A fetch-and-parse crawler does one thing with your HTML: it scans the response body for href attributes and queues the URLs it finds. It does not build a DOM and it does not wait for client-side rendering [14]. A menu entry that only becomes a URL after a client-side router runs gives that scan nothing to queue, and the destination is never requested, whatever is on it [13]. It looks fine in your browser, which is the part that makes it expensive to catch [21].
Search Engine Land's accounting is worth checking before you carry its numbers anywhere. The HTML group is given as 748 pages and the JavaScript group as 293, which sums to 1,041 against the 1,062 hierarchy pages the split is said to cover [15]. The 21-page gap matches the 21 top-level sections exactly, so the section landing pages are the likely residual [15]. Less tidy: Googlebot's phase-one coverage is reported as seven of 348 pages tracked in the JavaScript group, a group described elsewhere in the same write-up as 293 pages [20]. Someone changed a denominator mid-report. So read the ratios as directional. GPTBot's roughly 250 recovered pages is about 85% of a 293-page group [16], Bingbot's 212 about 72% [17], Googlebot's seven about 2% of its own larger denominator [18].
The zero is the load-bearing figure, and it is a figure about a specific setup. Sitemaps, breadcrumbs and every other discovery route were disabled, so each JavaScript section had exactly one way in [3]. For that zero to transfer to your site, the sections you care about would have to be single-pathed in the same way: no sitemap entry, no in-body cross-links from pages a crawler already has. The design isolates the navigation channel, which is what makes it a clean measurement and also what caps it. If you ship a complete sitemap, this result says your navigation contributes nothing to discovery, not that your pages are unreachable.
The phase-two split is what I would take to a review. GPTBot shows no renderer anywhere in these results and still came back within about two days of the markup changing [11]. Googlebot has a renderer, got further into the scripted hierarchy than any other crawler, and had found one converted page after 41 days [6][12]. Rendering capability and recrawl latency are separate properties of a crawler, and a plan that leans on the first is exposed to the second.
Adoption cost is low where the fix is markup: an anchor with a real href instead of a button with a handler, plus server-side rendering or prerendering wherever the framework injects navigation after load [14]. It gets expensive when the nav itself is fetched client-side after mount, because then the fix is a rendering-strategy change and not a component edit. My rule in this context is that navigation belongs in the server response and JavaScript decorates it. The report is careful on this point too: scripted navigation is not incompatible with discovery, it just cannot be the only path [19].
Ranked by verification strength, evidence, and original report placement.
A controlled 41-day experiment reported by Search Engine Land found that GPTBot and ClaudeBot discovered no hierarchy pages available only through JavaScript-injected links, while standard HTML navigation was crawled far more reliably.
In the first phase, GPTBot and ClaudeBot crawled all 748 pages in the HTML-linked group but found zero pages in the JavaScript-only hierarchy.
For Phase 2 the JavaScript links were converted to standard HTML links. GPTBot reswept the converted sections in about 48 hours and found approximately 250 new pages, Bingbot discovered roughly 212 new pages, and Meta External Agent added about 41.
In Search Engine Land's report, Vinicius Stanula describes a test site with roughly 2,400 pages across 21 top-level sections.
The experiment isolated navigation as the variable: sitemaps, breadcrumbs and other page-discovery routes were disabled, so to reach pages in the JavaScript group a bot had to render the parent page and process its JavaScript.
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 27, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
build
JavaScript navigation is now a discoverability decision, and 41 days of crawl logs show the cost1 distinct publisher
build
ChatGPT-User outfetched Googlebot for 34 days on one small site. Read your logs.1 distinct publisher
build
Cloudflare's one-click AI block names GPTBot, not the bot that decides if ChatGPT cites you1 distinct publisher
build
Reddit's ChatGPT citations fell 86% in four days, and the change was OpenAI's, not Reddit's1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Single secondary account of one unreplicated test, with internal numeric conflicts
All figures trace to one dev.to write-up of a Search Engine Land experiment; the primary report, crawl logs and methodology are not available in the supplied material. The design is reasonably specified (navigation isolated, alternative discovery paths disabled, two comparable groups) and the article states its own scope limit that crawl reach is not indexing. But the page counts do not reconcile — the group sizes sum 21 short of the stated 1,062 total and the JavaScript group is given as both 293 and 348 pages — and there is a single test site, one run, and no independent replication or crawler-operator response.
No usage or deployment data supplied
The cluster contains crawler-behaviour measurements from one synthetic test site, not adoption facts. There is nothing on how many production sites depend on JavaScript-only navigation, how many teams have applied the recommended server-side rendering or raw-HTML anchor fix, or any release, pricing or usage disclosure from the crawler operators. Rating adoption from a single-site benchmark would be inference rather than measurement.
Result is real but generalised beyond one deliberately worst-case test
The specific claim — GPTBot and ClaudeBot reached none of the JavaScript-only pages — is stated plainly and the article does carry the right caveats that crawling is not indexing and that JavaScript navigation is not inherently incompatible with SEO. The overstatement is in scope rather than substance: sitemaps and breadcrumbs were switched off to create a worst case, the sample is one test site, the JavaScript-group denominator is inconsistent, and the piece closes with a commercial visibility-scan pitch that treats the finding as a settled foundation for buying an audit. Positive but modest.
Findings framed as the setup for the publisher's own AI-visibility product pitch
The write-up ends with an explicit commercial call-to-action — Scalevise offering an 'AI Visibility and GEO Checker' scan to turn discoverability gaps into an improvement plan — placed directly after the technical conclusion, with no disclosure separating the promotion from the reporting. That aligns the article's incentive with maximising perceived AI-crawler risk. Offsetting factors: the underlying experiment is attributed to a third party (Vinicius Stanula via Search Engine Land) rather than run by the promoter, and the piece retains caveats that cut against maximal alarm.
Directionally plausible, numerically shaky, entirely single-sourced
The direction of the finding is coherent with the stated mechanism — crawlers that do not execute scripts cannot follow links that only exist after script execution — and the Phase 2 reversal (fast GPTBot and Bingbot re-sweeps once anchors were plain HTML) is an internally consistent control. Confidence is held down by one publisher, no primary-source access, no replication, unresolved page-count conflicts, and no adoption or outcome data beyond crawl reach.