Robots.txt Generator
Pick your rules, sitemap addresses and AI crawler permissions, then download your robots.txt file.
This tool builds the robots.txt file that tells search engines and AI crawlers which folders they may crawl, entirely in your browser; start from a ready template or add your own Allow and Disallow lines to each User-agent group. AI crawlers such as GPTBot, ClaudeBot and PerplexityBot get their own section where you can allow, block or leave each one untouched, with shortcuts for allowing everything or blocking only training crawlers in one click. Nothing you enter is ever sent to a server, the file is built entirely in your browser.
The rules you build appear below in a live preview; under each User-agent group, Allow and Disallow lines are listed in order, and the most specific path usually wins. Disallowing a path always requires it to start with a slash, otherwise crawlers may ignore the rule entirely; writing Disallow: / under User-agent: * removes your whole site from crawling, so the tool flags that line with a separate warning. Sitemap lines must be full addresses starting with https://, a relative path isn't accepted.
This tool only builds the file, it doesn't upload it anywhere; you need to copy the generated text and place it yourself at your site's root as robots.txt. A Crawl-delay line can be added but Google ignores that directive entirely, only some other crawlers respect it. robots.txt is a request, not a lock: badly behaved bots that ignore the rules can still disregard what's written here, so pages that genuinely need to stay hidden require password protection or server-level access control instead.
FAQ
How do I add the generated file to my site?
Save the copied or downloaded text as robots.txt and upload it to your site's root, so it's reachable at yourdomain.com/robots.txt.
What happens if I write Disallow: /?
That line removes your entire site from crawling for that User-agent group; the tool flags it with a separate warning, because it often hides the whole site from search results by mistake.
If I block AI crawlers, will my site still show up on Google?
Yes, blocking AI training crawlers such as Google-Extended doesn't affect Googlebot's search crawling; they are separate User-agents, and the shortcut in this tool respects that distinction.
Is anything I enter stored anywhere?
No, this tool runs entirely in your browser; no rule or address you enter is ever sent to a server or stored anywhere.
