Loading today’s stories
Build1 distinct publisher3 min readUpdated
A developer read the current Terms of Service for every feed in his news digest. The clauses that would break a summarization pipeline clustered in VC-data sites and well-lawyered corporate blogs.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
A developer who runs devdigest, a daily tech digest built on RSS, read the actual current Terms of Service for every source in his pipeline before charging money for it, quoting the operative clause or recording "no ToS found" [s1c3]. His finding, published on dev.to, is that the restrictive clauses cluster in VC-data companies and large corporate infrastructure blogs rather than in small publications [s1c4] - which is the opposite of the assumption most people building on feeds are working from.
The pipeline in question is deliberately modest: pull RSS, have a model read each excerpt to categorise and score it, write an original summary, email subscribers titles, summaries and links back to the publisher [s1c1]. Full article text is never reproduced or stored beyond a short-lived per-run cache, and reading anything requires a click through to the source [s1c2].
The clauses that break that design, per his reading: Crunchbase News bans using content "to train models (including generative artificial intelligence technologies)" and separately bans anything that "'Crawls,' 'scrapes,' or 'spiders'" the content [s1c5]. HPCwire/AIwire bans any "robot, spider, or other automatic device" without prior written permission, plus "any form of data extraction or data mining, or other commercial exploitation of any kind" [s1c6]. TechRadar, owned by Future plc, prohibits text or data mining and web scraping "for any purpose, including the development, training, fine-tuning or validation of AI systems or models" [s1c7]. Cloudflare's blog bars automated bots from scraping or data mining content "for developing, training, fine-tuning, or otherwise contributing to or improving" a machine learning model or AI system [s1c8]. Sifted, the European startup publication, carries near-identical language at section 7.6 [s1c9].
The permissive end reads like a list of places you would expect to be precious. InfoQ's terms state: "We permit the posting of a summary and then a link back to the InfoQ landing page" [s1c10]. MIT News says in its Terms of Use that it "offers RSS feeds for syndication purposes" [s1c11]. GitHub's terms say they "do not restrict lawful access to or use of the contents of public repositories by third parties" [s1c12]. arXiv's API terms name RSS-based discovery and notification tools as a permitted use case [s1c13]. TechCrunch maintains dedicated RSS terms, separate from its general ToS, explicitly permitting display of feed content with attribution and a link [s1c14]. Five named sources on each side of the line [s1c19].
His explanation is a guess, and he labels it as one: the restrictive sites have a data-licensing business to protect or expect to have one, so an AI-training ban is an asset to be sold later, while the permissive ones live on distribution and treat the feed as a front door [s1c15].
The more operationally useful part is that the obvious check fails in more than one direction [s1c20]. He originally cleared Towards Data Science because it is Medium-hosted and Medium's ToS contain no RSS, scraping, commercial-use or AI-training restriction [s1c16]. Medium's robots.txt disallows ClaudeBot, GPTBot and every other major AI crawler by name; his pipeline runs on an OpenAI model, so he removed the source [s1c17]. In the other direction, Sifted surfaces an RSS feed link on its own homepage while its terms prohibit both data mining and any automated "robot", "bot", "spider" or "scraper" [s1c18]. A published feed is not a licence, and a clean ToS read is a clean answer to the wrong question [s1c20].
What to watch: whether the AI-training clause becomes standard boilerplate at the large commercial publishers, in which case the safest sources for a summarization pipeline are university news offices, engineering blogs and syndication-native sites - the ones with no licensing revenue to defend [s1c15].
Follow any of these and your For You feed starts watching them — no settings page required.
Ranked by verification strength, evidence, and original report placement.
He expected restrictiveness to track company size, assuming big corporate sites with their own RSS feeds, press APIs and developer-relations teams would be relaxed and small scrappy publications would be strict; he found the most restrictive terms came from VC-data companies and large corporate infrastructure blogs.
Sifted surfaces an RSS feed link on its own homepage while its Terms of Use ban text and data mining and separately define and prohibit any "'robot', 'bot', 'spider', 'scraper' or other automated device."
The developer wrote that the obvious check is wrong in three different directions, and that checking the terms and not robots.txt gave him a clean answer to the wrong question; he also states that a published feed is not a licence.
The developer's product, devdigest, is a daily tech digest that pulls from RSS feeds, has a model read each excerpt to categorise and score it, writes an original summary, and emails subscribers a list of titles, summaries and links back to the original articles.
Full article text is never reproduced or stored beyond a short-lived per-run cache; to read anything, the subscriber clicks through to the publisher.
Before charging money for the digest, the developer read the actual current Terms of Service for all of his sources, not privacy policies or summaries, quoting the clause or explicitly recording "no ToS found."
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Verbatim primary quotes, single unverified reader
The load-bearing material is verbatim clause text from ten named publishers' terms, including a section reference for Sifted, which is checkable and unusually specific for a practitioner post. It is weakened by being one person's reading with no capture dates, no URLs for the quoted terms, no legal review, no publisher confirmation, and no published tally of the full audited source set.
One operator's own pipeline
The only observed uptake is the author's own digest: a disclosed production pipeline and one concrete source-list change driven by robots.txt. No other team, product, or publisher is shown adopting the audit practice, and no subscriber, revenue, or usage numbers are given.
Headline generalises a ten-source sample
The framing that AI-training bans live on big infrastructure blogs rather than small publications is a category generalisation from ten enumerated sources selected by one developer, and the causal explanation is self-labelled a best guess. The post itself is notably hedged - it rejects inferring bans from silence and warns about false positives - so the overstatement is modest and sits mostly in the headline-level pattern rather than in the underlying quotes.
Builder documenting his own paid product
The sole author is the owner of devdigest and states the audit was done 'before charging money for that', so the post doubles as a public compliance and legitimacy narrative for a product he is about to monetise, published on his own dev.to account. Offsetting this, the disclosed decisions cut against his interest - he dropped a working source and warns against the convenient reading of a published feed as permission - and he attributes a commercial motive to the publishers he quotes without evidence.
Checkable quotes, one self-interested source
Confidence is moderate: the clause-level facts are quoted verbatim and easy to verify, and the operational lesson about robots.txt versus terms stands on its own. It is held down by single-source provenance from the product's own builder, absent capture dates for terms that change frequently, no legal assessment of enforceability, and adoption limited to one pipeline.