Three files and tags are often mentioned together as "SEO settings": robots.txt, the noindex directive and sitemap.xml. They answer three different questions, and using one for another's job is behind many sites that either vanish from search or show pages they wanted hidden. In short:
| Question it answers | Read by | What it cannot do | |
|---|---|---|---|
robots.txt | May a crawler fetch this URL? | Crawlers, before fetching | Remove a page from results |
noindex | May this page appear in results? | Crawlers, after fetching the page | Work on a page that is blocked from fetching |
sitemap.xml | Which URLs exist and when did they change? | Search engines, as a hint | Force crawling or indexing |
You can try every example below in the robots.txt and sitemap generator; the results quoted are what its RFC 9309 tester returns.
robots.txt: a fetch policy, matched by longest rule
robots.txt lives at the root of the host (https://example.com/robots.txt) and applies only to that host and protocol. Take this file:
User-agent: *
Disallow: /search
Allow: /search/help
Disallow: /*?sort=
User-agent: Googlebot-Image
Disallow: /private-photos/What a crawler does with it:
| Crawler | URL | Result | Deciding rule |
|---|---|---|---|
| Googlebot | /search?q=shoes | blocked | line 2, Disallow: /search |
| Googlebot | /search/help | allowed | line 3, Allow: /search/help (longer) |
| Googlebot | /products?sort=price | blocked | line 4, Disallow: /*?sort= |
| Googlebot | /products?color=red&sort=price | allowed | no rule matches |
| Bingbot | /searching-tips | blocked | line 2, Disallow: /search |
| Googlebot-Image | /search?q=x | allowed | its own group has no such rule |
Three of those surprise people:
- Rules are prefixes.
Disallow: /searchalso blocks/searching-tipsand/search-engine-guide. Write/search/or/search$if that is what you mean. - `?sort=` is not `&sort=`. The parameter in second position is not matched. Add
Disallow: /*&sort=as well. - *A crawler with its own group ignores the `
group.** Googlebot-Image follows only lines 6 and 7, so it may fetch/search`. Repeat shared rules in every named group.
Since RFC 9309 (2022) these rules are standard: the longest matching rule wins, Allow wins a tie, * matches anything, $ anchors the end, and user-agent names match without regard to case.
noindex: the only reliable "keep this out of results"
noindex is a meta tag in the page or an HTTP header:
<meta name="robots" content="noindex" />X-Robots-Tag: noindexThe header works for PDFs and images, which have no <head>. The crawler must fetch the page to see either one. That is the classic trap: a page is blocked in robots.txt and given noindex. The crawler never fetches it, never sees the tag, and if other sites link to the URL, it can stay in results as a bare link with "No information is available for this page". To remove a page: allow crawling, add noindex, wait until it drops out, and only then block it if you must.
For one-off urgent removals, Search Console's removal tool hides a URL for about six months, but it is a temporary measure, not a replacement for noindex.
sitemap.xml: a list of suggestions
A sitemap tells search engines which URLs you consider canonical and, with <lastmod>, when they changed. It does not make anything rank or get indexed; it helps discovery, especially for new sites, large sites and pages with few links. The protocol limits each file to 50,000 URLs and 50 MB uncompressed; larger sites split into several files listed by a sitemap index. Google ignores <changefreq> and <priority> and uses <lastmod> only when it is consistently accurate, so a date that changes on every build teaches it to ignore your dates.
Two details often go wrong:
- Escaping.
https://example.com/blog/cà-phê?tag=a&page=2must be written as<loc>https://example.com/blog/c%C3%A0-ph%C3%AA?tag=a&page=2</loc>: percent-encoding for the URL, then&for XML. An unescaped&makes the whole file invalid. - Listing what you exclude. A URL in the sitemap that is blocked by robots.txt, has
noindex, redirects or canonicalises elsewhere sends mixed signals. A sitemap should list only final, indexable URLs.
Which one to use
- Keep crawlers out of infinite or expensive URL spaces (internal search, filters, calendars): robots.txt.
- Keep a page out of results (thank-you pages, staging copies, thin tag pages): noindex, with crawling allowed.
- Keep private content private: authentication. robots.txt is public and only a request; listing
/secret-plans/in it advertises the path. - Help new or deep pages get found: sitemap.xml, submitted in Search Console or referenced with a
Sitemap:line in robots.txt. - Opt out of AI training: named groups for crawlers such as GPTBot, ClaudeBot or Google-Extended in robots.txt. Google-Extended is only a token; it does not affect Google Search.
Common mistakes
- `Disallow: /` copied from a staging site to production. The single most expensive line in SEO. Test the live file after every deploy.
- Blocking CSS and JavaScript. Google renders pages; blocked resources can make a page look broken or empty to it.
- Expecting `Crawl-delay` to slow Google down. Google ignores it; it was never part of the standard.
- Using robots.txt to remove pages. Use noindex, then optionally block once they are gone.
- A sitemap on another host.
cdn.example.com/sitemap.xmlcannot listwww.example.comURLs unless both are verified together.
Try it
Paste your live robots.txt into the tester tab of the robots.txt and sitemap generator, list the URLs you care about, and check them for Googlebot and for the AI crawlers you intend to block. Then generate the sitemap from your URL list and check the row report for duplicates and stray hosts. For the tags inside each page, including noindex, use the Open Graph and meta tag generator.