Loading today’s stories
Build1 distinct publisher3 min readUpdated
A dev.to write-up documents the failure modes of scraping Trends' internal endpoints. The worst one is not an error at all: it is a successful-looking response carrying nothing.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
Google Trends has no public API, only the internal JSON endpoints that trends.google.com's own single-page app calls, and according to a dev.to write-up on building a client against them, those endpoints reply with HTTP 200 and an empty body when Google decides the caller looks like a bot [1][2]. No 429 is sent [2]. Any client that branches on status code alone will therefore log a block as a legitimate empty result, and that emptiness propagates into whatever you built on top of it.
The remedy in the post is unglamorous: check `resp.text.strip()` on every call that is supposed to return a payload, and if it is empty, rotate the proxy session and retry exactly as you would on a 429, because functionally that is what it is [3]. The cost of not doing this is not a crash. It is a chart with a hole in it that nobody notices for a month.
The same document collects three more failures with the same shape, where a broken client looks like a boring answer. Every Trends JSON endpoint prepends an XSSI-protection prefix, so naive `json.loads(resp.text)` raises `JSONDecodeError` on a perfectly healthy response [4]. The prefix is not consistent either: `/trends/api/explore` sends the comma-free variant while some widgetdata endpoints send the version with a comma, which means the longer form has to be tested first or a short-prefix response gets mis-stripped [5]. Related queries and related topics return the literal string `"Breakout"` instead of a number when a term's volume grows more than 5,000% in the window, so a schema typed as `value: int` either crashes or coerces the row to `None` precisely when the row is interesting [8]. And `/trends/api/explore` accepts at most five keywords per comparison; submit a sixth and you get a rejected `req` payload rather than an error that explains itself [9].
The endpoint rename is the clearest illustration. `/trends/api/widgetdata/multiGeo`, the path every geo-breakdown request used, began returning 404 with no deprecation notice; the replacement is `/trends/api/widgetdata/comparedgeo`, confirmed in the author's code comment on 2026-06-09 [6]. The author caught it as a failure cluster in run logs on the `interest_by_region` widget, and notes that a scraper built once and forgotten simply eats that as "no regional interest" rather than "broken client" [7].
The throughput arithmetic follows from the five-keyword ceiling. The client takes up to 100 keywords, splits them into batches of five, and sleeps two seconds between batches so consecutive `/explore` calls do not read as a burst [10]. That is 20 requests for a full submission [1] and 38 seconds of enforced idling before any network time [2]. Retry behaviour adds more: Chrome, Firefox and Safari TLS fingerprints rotated via curl-cffi, residential proxy exit IPs rotated on every soft block, and backoff of 2, 4, 8, 16 and 30 seconds across up to five attempts [11], which is 60 seconds of waiting before a single request is abandoned [3]. The author's stated justification for the whole exercise is that Trends data sits behind no login wall and no paywall on Google's own site [12].
What to watch: instrument the empty-body rate as a first-class metric rather than a log line, because it is the only signal that distinguishes throttling from genuine absence of data. Watch 404s per endpoint too. The `multiGeo` rename arrived unannounced [6], and the next one will as well.
Follow any of these and your For You feed starts watching them — no settings page required.
Ranked by verification strength, evidence, and original report placement.
Google Trends has no public API; what exists are the same internal JSON endpoints that the trends.google.com single-page app calls.
Google Trends widgetdata endpoints answer with HTTP 200 and an empty body when Google decides the client looks like a bot; the request is soft-blocked without sending a 429.
The recommended fix is to stop trusting the status code alone: check resp.text.strip() on every call that should return a payload, and if it is empty, rotate the proxy session and retry exactly as on a 429.
Every Trends JSON endpoint prepends an XSSI-protection prefix before the JSON body, so naive json.loads(resp.text) throws a JSONDecodeError on a perfectly healthy response.
The XSSI prefix is inconsistent: /trends/api/explore sends the variant without a comma, some widgetdata endpoints send the variant with a comma, so the longer variant must be checked first to avoid mis-stripping a short-prefix response.
/trends/api/widgetdata/multiGeo, the endpoint every geo-breakdown request used, started returning HTTP 404 with no deprecation notice; it is now /trends/api/widgetdata/comparedgeo, per a code comment stating it was confirmed 2026-06-09.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Concrete and self-consistent, but single-source and unverified
The claims are unusually specific and internally checkable — named endpoints, two literal XSSI prefixes, KEYWORD_BATCH_SIZE = 5, an explicit backoff ladder, a dated code comment — and are shown as code rather than asserted. Against that, everything rests on one self-interested vendor post with no second observer, no Google statement, and no reproducible artifact (repo, run log, or dataset) a reader could independently exercise.
No adoption evidence beyond the author's own product
The only adoption-adjacent facts are the author's own run-log anecdote and their promoted scraper with a price list. There are no user counts, downloads, customers, third-party deployments, or corroborating reports of the same soft-block or endpoint-rename behavior, so the extent to which anyone else relies on these patterns cannot be measured from the supplied sources.
Slightly overstated by product framing, technical core is proportionate
The technical claims are narrow, falsifiable and matched to the code shown, so there is little inflation in substance. The mild overstatement comes from packaging: a general-sounding 'Google Trends API' framing around undocumented endpoints, an unqualified assertion that scraping stays inside a legitimate public surface because the data is free, and a sales close that positions the vendor's Actor as the answer without any comparative or durability evidence.
Explicit commercial incentive: the post sells the fix
The author is the vendor of a paid Google Trends Scraper and closes with per-run and per-row pricing, so the piece functions as capability marketing for the very failure modes it documents. That incentive favors emphasizing hard-to-solve breakage and downplaying legal, durability and data-completeness caveats, though the incentive is disclosed openly rather than hidden.
Moderate-low: plausible and specific, one interested witness
Confidence is held down by structural thinness — one publisher, one article, no corroboration or contradiction available — and by the vendor's stake in the narrative. It is held up by the granularity of the technical detail, the fact that the mechanisms described (XSSI prefixes, bot-detection soft blocks, undocumented endpoint churn) are ordinary for internal Google endpoints, and by derived arithmetic that checks out against the disclosed constants.
build
A 5x publishing increase cost one site 1,000 indexed pages and every impression1 distinct publisher
build
Before you spend quota on an agent skill, make it pass an eval harness1 distinct publisher
build
Geofencing beats GPS polling on power, then loses to the OEM battery optimiser1 distinct publisher
build
The eBay scraper defect that bills your customer: sponsored cards in organic markup1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 21, 2026