How to use Robots.txt & sitemap generator
- 1
Pick a starting point
Choose a preset such as Block AI training, keep AI search, or start from Allow everything and edit the groups.
- 2
Add groups and rules
For each group pick the crawlers, from Googlebot to GPTBot and ClaudeBot, then add Allow and Disallow paths and your sitemap URL.
- 3
Test real URLs
Open Tester, paste any URL or path and see which rule decides for each crawler, using the same longest-match logic Google uses.
- 4
Build the sitemap
Paste your URLs or let our server crawl your site, then download sitemap.xml, or a ZIP with an index when you have more than 50,000 URLs.
Limits and Pro
Free: crawls of up to 300 pages with 3 parallel requests, 10 seconds per page and 90 seconds per crawl, 3 crawls every 10 minutes; pasted lists of up to 200,000 URLs; sitemap files split at 50,000 URLs or 50 MB.
About Robots.txt & sitemap generator
This generator covers the two files that tell crawlers what to fetch. The robots.txt builder writes groups of User-agent lines with Allow and Disallow rules, optional Crawl-delay values and Sitemap lines. Its crawler list includes search engines and the AI agents site owners ask about most: OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, Anthropic's ClaudeBot, Claude-SearchBot and Claude-User, PerplexityBot, and control tokens such as Google-Extended and Applebot-Extended, each labelled as training, search or user-initiated. A side panel shows at a glance which crawlers may fetch your home page.
The tester applies RFC 9309, the Robots Exclusion Protocol standard: a crawler follows the group naming it, or the * group; the longest matching pattern wins, Allow wins a tie, * matches anything and $ anchors the end of the URL. It works on the builder's rules or on any robots.txt you paste, and lists lines crawlers will ignore.
The sitemap generator writes sitemaps.org XML with escaped locations and optional lastmod dates, removes duplicates and splits large lists into files of 50,000 URLs with a sitemap index. Its crawler follows same-host links, obeys robots.txt and noindex, and reads lastmod from page metadata. robots.txt is advice that polite crawlers follow, not access control: it cannot hide pages from people or from crawlers that ignore it.
Privacy
The robots.txt builder, the tester and sitemaps made from a pasted list all run in your browser and send nothing. Only Crawl my site uses our server: it receives the start address, fetches your public pages and robots.txt, and streams the URLs it finds back to you without storing them.
Features
AI crawler aware
21 named crawler tokens, from Googlebot to GPTBot, ClaudeBot and PerplexityBot, labelled by purpose.
RFC 9309 tester
Longest-match verdicts with the deciding line for any URL and any crawler.
Sitemaps that scale
Paste or crawl, lastmod dates, deduplication and automatic index files above 50,000 URLs.