Build1 publisher3 min readPublished
A trailing slash in robots.txt would have left pub-trivia.app's dashboard crawlable
Pub-trivia.app's developer warns that Disallow: /dashboard/ would block every dashboard page except the bare /dashboard its homepage footer links to. Blocking a fetch never stops indexing, so the noindex tag has to go live before the Disallow.
The Engineer · Build desk
Drafted by a language model from the sources cited here and checked against its claim ledger before publication. How we use AISend a correction
What happened
- Pub-trivia.app generates its robots.txt from a single array of seven private paths, including /dashboard, /api and /auth, and none of them ends in a slash.
- Player URLs under /play are printed on table cards and shared from phone to phone in pubs, so the developer treats them as crawled regardless and noindexes the whole segment.
- Preview and staging builds serve the same pages as production, and an indexed one would put a second copy of all seventy pages in the index, competing with the first.
- Login, signup and password-reset pages were removed from the sitemap but left crawlable, because a sitemap lists pages meant to rank and robots.txt only governs fetching.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- exposure A trailing-slash rule passes a spot check on any subpage while leaving the segment's root fetchable, so a robots.txt audit has to request the bare path itself.
- decision Taking an already-indexed segment private becomes two deploys separated by a recrawl, because a noindex shipped alongside the Disallow is never fetched.
- constraint The slash-less form closes the root-page gap but also blocks any public path sharing the prefix, so short entries have to be checked against the public route list.
The rule underneath is plain string comparison, according to the post's author. A crawler checks whether the request path begins with the Disallow value [2]. `/dashboard/settings` and `/dashboard/billing` begin with `/dashboard/`, so they are blocked. The bare `/dashboard` does not, so it stays crawlable [2]. On pub-trivia.app that bare path is linked from the footer of the public homepage. It is the most discoverable URL in the whole private area [3]. "A trailing slash would have blocked everything except the one page a crawler was guaranteed to find," the author wrote [4].
The directive is not a glob either, the post says. `Disallow: /api` and `Disallow: /api/` are different instructions, and the shorter one is almost always the intended one [5]. Prefix matching cuts both ways, so I'd check every short entry against the public routes. `Disallow: /api` also blocks `/api-docs`, `/apiary`, or any other path that starts with those four characters [1].
The bigger correction is about what Disallow does at all. It tells a crawler not to fetch a path. It does not keep the path out of the index [6]. A URL Google has seen in a link, in a sitemap or pasted into a forum can be indexed as a title-only result the crawler never read [6]. So the private segments repeat the instruction in page metadata, `robots: { index: false, follow: false }` [7].
The two signals interfere. Once a path is disallowed, Google cannot fetch the page, so it never sees a noindex added afterwards [8]. The author's fix is a sequence. Ship the noindex, leave the page crawlable until it has been seen, then add the Disallow to stop spending crawl budget on it [9]. Adding both in one commit to a segment that is already indexed does the opposite of what was intended, and nothing reports it [9].
Placement matters too. I think the layout is the right home for the noindex. One declaration in `app/dashboard/layout.tsx` covers every present and future route under `/dashboard` [10]. The alternative is appending each new private route to the array in `app/robots.ts`, and forgetting one produces no error and no failing test [10].
Staging applies both signals at once. Non-production builds return `disallow: "/"` for every user agent and set the root `index` and `follow` values to `IS_PRODUCTION_SITE` [13]. That flag is one comparison, `SITE_URL === 'https://pub-trivia.app'`, resolved once in a module that also feeds `metadataBase`, the sitemap, Stripe redirect URLs, password reset emails and the QR codes on table cards [14]. That single origin is the best engineering in the post. According to the author, deriving it separately in each caller is how a deploy ends up serving a sitemap for one host and sending emails for another [14]. The ordering rule applies here as well. A crawler that obeys the blanket Disallow never fetches a preview page, so it never reads the noindex. A preview URL that leaks into a public link could still be indexed title-only [2].
What to watch
- Search Console coverage for pub-trivia.app's private segments, showing whether the noindex was read before any Disallow took effect.
- Any pub-trivia.app preview or staging URL appearing in search results, which would test whether the root noindex does anything behind a blanket Disallow.
- A new private route added outside a noindexed layout, the failure the post says produces no error and no failing test.