Eight names in Medium's robots.txt close training ingestion while retrieval stays open

A robots.txt file is matched by user-agent, and a named group overrides the wildcard. Medium's file has the usual `*` rules and then a second group listing eight agents ^[1]. That group gets `Disallow: /` plus seven `Allow` lines, and all seven are business pages ^[2]. A crawler in that group can reach the membership page; a member's posts are disallowed ^[21].

"AI crawler" covers two jobs. The file closes one of them ^[4]. The eight names are the training and bulk-collection side: GPTBot for OpenAI, ClaudeBot for Anthropic, Applebot-Extended for Apple Intelligence, meta-externalagent for Meta, Bytespider for ByteDance, Amazonbot ^[5]. The retrieval crawlers behind AI search fall outside that group, so the `*` rules cover them and they are allowed ^[6]. According to the dev.to post, a Medium post can therefore still turn up as a cited source in Perplexity or an AI Overview, because it reaches those engines through a search index and those indexes are open ^[7].

The group leaves out two names. Google-Extended crawls nothing; it is the token that controls whether content Google has already crawled may be used for Gemini training. Leaving it unblocked keeps that path open while OpenAI's and Anthropic's equivalents are shut ^[8]. "I don't read this as a loophole anyone left on purpose," the author wrote, describing the file as the normal state of a hand-maintained list of names, always slightly behind a user-agent landscape that adds a new one every few months ^[10][11].

A model reading your own domain treats what it finds as self-description, and the same sentence on a domain you do not control is read differently ^[13]. The author's case is a Gemini comparison prompt that described his tool as a "Local SEO & AI Indexing" product "built on top of Scrapeless", and no such product sits underneath it ^[14]. He attributes the error to the bare acronym: a lot of indexed repositories use GEO and "geographic" interchangeably. A model with no nearby context can resolve it to the geographic sense and attach whatever vendor sits in that space ^[15]. The fix was to put the expansion next to the name in the hero paragraph, the meta description, and the `WebSite` and `WebApplication` nodes of the JSON-LD graph ^[16].

On the built HTML, the entity-disambiguation check went from 1/3 to 3/3 and the citability score from 60 to 62 ^[17]. The two-point gain sits on a scale the post does not define ^[22][23]. For those numbers to say anything about another site you would need the same checker and a page whose entity name was ambiguous the same way. The 1/3 to 3/3 is the more legible half: three named checks that pass after the edit. And the fix applies only to pages the author owns ^[18].

robots.txt is a request. It carries no enforcement. So the file tells you a platform's stated position, and what every crawler does with it is a separate question ^[19]. The author's conclusion is that Medium is fine for human readers and distribution, and that corroboration a model can crawl needs a platform that allows the crawlers. He published the column on dev.to ^[20].