Screaming Frog could have crawled Carrefour's 30,000 pages if the edge had let it in

Sustained concurrency from one desktop, on one IP, with one user agent is the request pattern adaptive rate limiting exists to catch. Slowing down and changing the user agent are available moves, and the post says they are not the reliable ones ^[11]. An audit tool whose coverage depends on the retailer's edge team agreeing to add an allowlist entry is a permissions product first and a software product second.

Thirty thousand pages is well inside what the tool handles. Screaming Frog's memory storage mode is recommended up to 500k URLs and handles a couple hundred thousand on 8GB of RAM, and database storage mode goes into the millions ^[5]. "Anyone claiming 30k pages broke the tool is describing a misconfiguration, not a limit," the author wrote ^[6].

Rendering is a time cost, and the time is checkable. The SEO Spider renders with Chromium, resource crawling has to stay enabled, and Screaming Frog's own guidance notes most servers do not want to be crawled faster than about 5 URLs per second ^[7]. Thirty thousand URLs at five per second is 6,000 seconds, or one hour and forty minutes ^[1]. The post calls 30k rendered URLs a multi-hour job on a laptop ^[8]. For both figures to hold, the effective page rate has to sit below five per second. Fetching each page's scripts and API responses on the same politeness budget does that.

The six hours come from the URL space. A faceted grocery catalog is a combinatorial space. The exclude regex that contains it has to be written from facet patterns you learn by crawling, and on an unfamiliar site the first crawl diverges and you find out at hour six ^[9]. Six hours at five requests per second is 108,000 requests, 3.6 times the nominal page count ^[2].

The post's title is "Googlebot Renders Your JavaScript. ChatGPT Doesn't." ^[13] The storefront it describes is client-rendered: the initial HTML response is a shell, with product, category and facet content assembled in the browser ^[3]. The run itself reports a desktop crawler and server-side log analysis, and no fetch by an LLM crawler ^[14]. The author wrote that the honest framing is not crawling harder; it is starting from the server side, where you do not need to be let in because you are already inside ^[12]. The post's access argument holds on its own evidence. Measuring what a model's fetcher gets back from shell HTML is a different exercise, and this run did not do it.

Suppose the crawl had finished. The post's argument is that a 30,000-SKU catalog rarely carries more than a few hundred URLs with meaningful non-branded demand, almost all of them category, subcategory and brand pages ^[15]. Read "a few hundred" as 300 and that is 1 percent of the crawlable pages ^[3]. The decisions sit in that 1 percent: which facet subset gets promoted to a real landing page with its own copy, and what happens to the thousands of product pages that die every month. That last one is a choice between 404 and 410, a redirect to the category, or staying live with alternatives ^[16][18]. Weekly flyer and campaign pages go stale into orphans and soft 404s, and the post says those are often the pages that earned the site its few real backlinks ^[19].