Three tabs, all working on your device:
- robots.txt builder: groups of user agents with Allow and Disallow rules, an optional Crawl-delay and Sitemap lines, with presets for allowing or blocking everything, WordPress, and blocking AI crawlers. Warnings appear as you type: a path without a leading slash, a sitemap URL that is not absolute, an agent name with characters crawlers ignore.
- robots.txt tester: paste any robots.txt (or use the one you just built), name a crawler such as Googlebot or GPTBot, and list URLs. Each one is marked allowed or blocked, with the line number of the rule that decided it.
- sitemap.xml: paste URLs, one per line, optionally with a date after a comma. You get a valid sitemap, split into several files plus a sitemap index when there are more than 50,000 URLs or a file would pass 50 MB, and a report of the rows that were left out and why.
How the tester decides, per RFC 9309
robots.txt was standardised as RFC 9309 in September 2022, and Google's crawler follows it. The tester applies the same rules:
- Pick the group. The crawler's product token is compared with every
User-agent line, ignoring case. All groups that name it are merged. If none does, the * group is used; if there is no * group either, everything is allowed. A crawler with its own group does not also follow *. - Find matching rules.
* matches any run of characters and $ at the end means "the URL ends here", so /*.pdf$ blocks /files/a.pdf but not /files/a.pdf?download=1. Matching is case-sensitive and starts at the beginning of the path. - The longest rule wins. With
Allow: /example/page/ and Disallow: /example/page/disallowed.gif, the GIF is blocked because its rule is longer. If an Allow and a Disallow are equally long, Allow wins. - Encoding is normalised.
/caf%C3%A9 and /café are the same path; %62%61%7A matches baz. %2A and %24 in a rule match a literal * and $ in the URL. - /robots.txt itself is always allowed.
The test suite checks these against the examples in RFC 9309 and in Google's documentation.
Blocking AI crawlers
The presets list user agents published by the companies that run them, as of 14 October 2026: training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent, Bytespider, Amazonbot and others) and, in the stricter preset, the crawlers that fetch pages for AI search answers or when a user asks a chatbot about a page (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User). Google-Extended and Applebot-Extended are opt-out tokens, not separate crawlers: Googlebot still indexes your pages for Search. Blocking the search crawlers keeps your pages out of those AI answers as well. robots.txt is a request, not a lock: some crawlers ignore it, and only a firewall rule actually stops a request.
Sitemaps: what matters and what does not
Each <loc> must be absolute, on the same host as the sitemap, under 2,048 characters, and escaped twice: percent-encoding for the URL (the tool turns /cà-phê into /c%C3%A0-ph%C3%AA) and XML entities for the file (& becomes &). Google ignores <changefreq> and <priority> and uses <lastmod> only when it is consistently accurate, so set a date only when it is the real date the content changed. For split files, give the address where you will publish them; the index lists them there. Submit the sitemap (or the index) in Search Console, or add a Sitemap: line to robots.txt.
The article robots.txt vs noindex vs sitemap explains which of the three to use when, including the classic mistake of blocking a page you want removed. To set the title and share image of each page, use the Open Graph and meta tag generator; to check a URL's encoding, the URL encoder.
