Three files, three contracts: robots.txt, sitemap.xml and llms.txt are not rivals

A dev.to explainer published under the webpixie byline makes a distinction operators keep collapsing: robots.txt, sitemap.xml and llms.txt do three different jobs, they are not interchangeable, and most comparisons of them blur the boundary ^[1]. That is worth restating because the failure modes point in opposite directions: one careless Disallow line starves your own retrieval, while a robots.txt written as a secrecy tool publishes the exact paths you wanted hidden ^[8].

robots.txt is access policy. It sits at /robots.txt, it tells well-behaved bots which paths they may fetch, and since 2022 it has had a formal specification in RFC 9309 after decades as de-facto convention ^[3][5]. The list of things it does not do is longer. It does not deindex: Disallow stops crawling, and content a search engine already knows about needs a noindex meta tag or HTTP header to leave the index ^[7]. It does not protect: the file is publicly readable, and authentication is what keeps content private ^[8]. It does not bind anyone unwilling to be bound, which leaves server-side blocking by user agent or IP as the only enforcement layer ^[9]. On JavaScript-heavy sites, disallowing the path to a JS file that indexed pages depend on can leave search engines confused about what those pages contain ^[10].

It is also not a single switch for AI policy. According to the explainer, blocking GPTBot stops OpenAI's training crawler while ChatGPT can still cite the site through a separate live-browsing bot, and other AI products use different bot names, so the relationship between training and retrieval is per vendor ^[11]. The post's minimum-viable file reflects that shape: a wildcard stanza blocking /admin/ and /api/internal/, a second stanza aimed only at GPTBot, and a Sitemap line ^[21]. That Sitemap directive is the one bridge between the two files; otherwise they are separate concerns ^[6].

sitemap.xml is the crawl map. It commonly sits at /sitemap.xml, follows the protocol defined at sitemaps.org, and can carry per-URL last-modified, priority and change-frequency hints ^[12]. Every entry is a suggestion rather than a command, with indexing decisions still turning on content quality, internal linking and other signals ^[13]. Pages nothing links to remain weak candidates whether or not they appear in the file ^[14], and refreshing every lastmod to today makes the file untrustworthy and is widely discounted by major search engines ^[15]. The format was designed for search-engine indexing, not LLM retrieval, though some AI crawlers do use sitemaps to discover URLs ^[16].

llms.txt is the reading brief and the youngest of the three: introduced in 2024 with its spec at llmstxt.org, a single markdown file at /llms.txt giving assistants a curated, structured view of your most important content ^[17]. The stated point is to spare the model from crawling and synthesizing the whole site every time someone asks about you ^[18]. It arrived two years after robots.txt got its RFC ^[19]. Note the verb in the explainer's own description: llms.txt is meant to brief AI assistants ^[2]. Nothing in the file compels anything, and neither do the other two on their own ^[20].

Watch whether teams keep the three contracts separate in operation: llms.txt used as access control, or robots.txt used as curation, are the same mistake in different directions. Watch, too, the vendor bot inventory, since a per-vendor split between training crawlers and live-browsing agents means a policy stance is a list of user agents, not a line in one file ^[11].