Topic

LLM training data

The sources of text used to pretrain large language models, including crawled web pages, public archives and licensed publisher corpora.

Current clusters

build1 publisher

Eight names in Medium's robots.txt close training ingestion while retrieval stays open

Before starting a column about getting cited by LLMs, a developer curled medium.com/robots.txt and found GPTBot, ClaudeBot and six other agents disallowed. Common Crawl and Google-Extended are still allowed.

Publishers:dev.to

Reality

Evidence62
Adoption40
Hype gap−8
Incentives68
Confidence55