Build1 publisher3 min readPublished
Three files, three contracts: robots.txt, sitemap.xml and llms.txt are not rivals
A dev.to explainer redraws the boundary most comparisons blur: access policy, crawl map, reading brief. Conflating them is how teams block themselves or expose what they meant to hide.
The Engineer · Build desk
Drafted by a language model from the sources cited here and checked against its claim ledger before publication. How we use AISend a correction
What happened
- A dev.to explainer titled "robots.txt vs llms.txt vs sitemap.xml: what each is for", published under the webpixie byline, argues the three files have three different jobs, are not interchangeable, and that most online comparisons blur the boundary between them.
- The explainer's framing: robots.txt is a fence that controls who is allowed to crawl, sitemap.xml is a map that tells search crawlers what is worth crawling, and llms.txt is a brief that is meant to brief AI assistants on what to read first about you.
- robots.txt sits at the root of a site at /robots.txt and tells well-behaved bots which paths they are allowed to fetch; it is an access policy expressed as a plain-text allowlist or denylist.
- Compliance with robots.txt is voluntary; malicious crawlers ignore it, while major search engines and the public AI crawlers respect it.
- The robots.txt protocol was standardized as RFC 9309 in 2022, formalizing decades of de-facto convention.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
A dev.to explainer published under the webpixie byline makes a distinction operators keep collapsing: robots.txt, sitemap.xml and llms.txt do three different jobs, they are not interchangeable, and most comparisons of them blur the boundary [1]. That is worth restating because the failure modes point in opposite directions: one careless Disallow line starves your own retrieval, while a robots.txt written as a secrecy tool publishes the exact paths you wanted hidden [8].
robots.txt is access policy. It sits at /robots.txt, it tells well-behaved bots which paths they may fetch, and since 2022 it has had a formal specification in RFC 9309 after decades as de-facto convention [3][5]. The list of things it does not do is longer. It does not deindex: Disallow stops crawling, and content a search engine already knows about needs a noindex meta tag or HTTP header to leave the index [7]. It does not protect: the file is publicly readable, and authentication is what keeps content private [8]. It does not bind anyone unwilling to be bound, which leaves server-side blocking by user agent or IP as the only enforcement layer [9]. On JavaScript-heavy sites, disallowing the path to a JS file that indexed pages depend on can leave search engines confused about what those pages contain [10].
It is also not a single switch for AI policy. According to the explainer, blocking GPTBot stops OpenAI's training crawler while ChatGPT can still cite the site through a separate live-browsing bot, and other AI products use different bot names, so the relationship between training and retrieval is per vendor [11]. The post's minimum-viable file reflects that shape: a wildcard stanza blocking /admin/ and /api/internal/, a second stanza aimed only at GPTBot, and a Sitemap line [21]. That Sitemap directive is the one bridge between the two files; otherwise they are separate concerns [6].
sitemap.xml is the crawl map. It commonly sits at /sitemap.xml, follows the protocol defined at sitemaps.org, and can carry per-URL last-modified, priority and change-frequency hints [12]. Every entry is a suggestion rather than a command, with indexing decisions still turning on content quality, internal linking and other signals [13]. Pages nothing links to remain weak candidates whether or not they appear in the file [14], and refreshing every lastmod to today makes the file untrustworthy and is widely discounted by major search engines [15]. The format was designed for search-engine indexing, not LLM retrieval, though some AI crawlers do use sitemaps to discover URLs [16].
llms.txt is the reading brief and the youngest of the three: introduced in 2024 with its spec at llmstxt.org, a single markdown file at /llms.txt giving assistants a curated, structured view of your most important content [17]. The stated point is to spare the model from crawling and synthesizing the whole site every time someone asks about you [18]. It arrived two years after robots.txt got its RFC [19]. Note the verb in the explainer's own description: llms.txt is meant to brief AI assistants [2]. Nothing in the file compels anything, and neither do the other two on their own [20].
Watch whether teams keep the three contracts separate in operation: llms.txt used as access control, or robots.txt used as curation, are the same mistake in different directions. Watch, too, the vendor bot inventory, since a per-vendor split between training crawlers and live-browsing agents means a policy stance is a list of user agents, not a line in one file [11].