8 min read · updated August 2, 2026
Robots.txt Generator: Get Crawl Rules Right First Time
Robots.txt Generator
Create robots.txt file — free, no signup
A robots.txt generator builds the small plain-text file that tells search engine crawlers which parts of your site to fetch and which to skip, without you having to remember the directive syntax or the parsing quirks. The file itself is deceptively simple, a handful of User-agent, Disallow, and Allow lines, but the rules for how crawlers interpret those lines are specific enough that hand-written files regularly do something other than what their author intended.
The stakes are asymmetric. A missing robots.txt costs you nothing, since crawlers treat its absence as permission to crawl everything. A wrong one can remove your site from search results: the single line Disallow: / left over from a staging deployment is one of the most reliably catastrophic typos in technical SEO. This guide explains how crawlers actually fetch and parse the file, how rule precedence and wildcards work, and the scenarios, from WordPress cleanups to blocking AI training crawlers, where a generated, validated file beats improvising syntax from memory.
What robots.txt does, and the security myth
The robots exclusion protocol dates to 1994 and was formalized as RFC 9309 in 2022. The file lives at exactly one place, the root of the host, as in example.com/robots.txt, and states which URL paths each crawler may request. Its real purposes are crawl management: keeping bots out of infinite URL spaces like faceted filters and calendars, keeping them focused on pages you care about, and cutting server load from aggressive crawlers.
What it is not is access control. Compliance is entirely voluntary; reputable crawlers like Googlebot and Bingbot honor the file, while scrapers and malicious bots simply ignore it. Worse, the file is public, so listing an admin path in Disallow advertises that path to anyone curious enough to read it. Anything genuinely private belongs behind authentication, not behind a polite request.
The other core distinction people miss: robots.txt controls crawling, not indexing. A blocked URL can still appear in Google results, shown with no description, if other sites link to it, because Google knows the URL exists without being allowed to fetch it. Keeping a page out of the index requires a noindex robots meta tag or header, and that tag only works if the crawler is allowed to fetch the page and see it. Blocking a page in robots.txt while also marking it noindex is self-defeating, since the block hides the noindex.
How crawlers fetch and parse the file
Before crawling a host, a compliant bot requests /robots.txt and caches the result, generally for up to 24 hours, which is why rule changes do not take effect instantly. The response code matters: a 404 means no restrictions, crawl everything, while persistent server errors are treated by Google, after retries, as if the file did not exist. Each protocol and subdomain combination needs its own file; the one on example.com does not cover shop.example.com.
Parsing works on groups. The bot scans for the User-agent line that best matches its own name, follows only that group's rules, and falls back to the wildcard asterisk group if no specific match exists. This is the detail that breaks the most hand-written files: groups do not inherit. If you write a general group blocking /admin/ and then add a specific Googlebot group to block one extra path, Googlebot obeys only its own group, and /admin/ is now open to it unless repeated there.
Within the matching group, every Disallow and Allow rule is compared against the URL path, and the most specific rule, meaning the one matching the longest number of characters, wins; on a genuine tie, the less restrictive Allow prevails under Google's implementation. Two wildcard operators exist: the asterisk matches any sequence of characters, and the dollar sign anchors a match to the end of the URL, so Disallow: /*.pdf$ blocks PDFs specifically. Paths are case sensitive, meaning /Admin/ and /admin/ are different strings to a crawler, and an empty Disallow line means the group blocks nothing. A generator encodes all of this so you declare intent instead of reciting parser behavior.
Directives that do what you intend
The patterns below cover nearly every real file. Rules are per-group, so remember to repeat shared rules inside any bot-specific group you add.
- Block a directory: Disallow: /admin/ inside a group stops matching bots fetching anything under that path. The trailing slash matters; without it you also block /admin-guide and any other path with that prefix.
- Carve an exception: Allow: /admin/public/ under the same group reopens one subtree of a blocked directory. The longest-match rule is what makes the exception beat the broader block.
- Block a URL pattern: Disallow: /*?sort= stops crawling of every sorted variant of listing pages, one of the highest-value rules on ecommerce sites where filter combinations multiply into millions of near-duplicate URLs.
- Block a file type: Disallow: /*.pdf$ uses the end anchor so only URLs ending in .pdf match, not paths that merely contain the string.
- Target one bot: a User-agent: GPTBot group with Disallow: / opts that crawler out of the whole site while leaving search crawlers untouched, since each bot obeys only its own best-matching group.
- Declare your sitemap: Sitemap: https://example.com/sitemap.xml stands outside any group, takes a full absolute URL, and helps engines discover pages even when they arrive via a different path.
- Know what to skip: Crawl-delay is honored by Bing but ignored entirely by Google, which paces itself from server response instead; do not expect it to slow Googlebot.
Scenarios worth writing rules for
Most sites need only a few lines, but the few lines differ by situation.
- WordPress: block /wp-admin/ while allowing /wp-admin/admin-ajax.php, the exception WordPress itself recommends because front-end features post to that endpoint. Do not block /wp-content/, since that is where the CSS and JavaScript live that Google needs to render pages.
- Ecommerce: keep crawlers out of cart, checkout, and account paths and out of parameter permutations from filters and sorting. On large catalogs this is crawl budget management with real consequences, redirecting the crawler's limited fetches from duplicate filter URLs toward products.
- Staging environments: password protection is the correct fix, with a Disallow: / file as backup. If a staging robots.txt does its job, remember it must not travel to production; keeping the two files different by deployment configuration, not by hand, prevents the classic accident.
- AI crawlers: bots like GPTBot, ClaudeBot, and CCBot identify themselves and honor robots.txt, so a per-bot group opts your content out of training data collection without affecting search visibility. This is a policy choice, and the mechanism is the same three lines either way.
- Documentation and internal search: block your site's internal search results pages, such as Disallow: /search, since crawling infinite query permutations wastes fetches and can leak odd queries into the index.
Mistakes that cost real traffic
The failure modes repeat, and almost all of them are silent until traffic drops. The staging block that ships to production is the canonical one: a bare Disallow: / under the wildcard group tells every search engine to stop fetching the site, and rankings decay as the index goes stale. The reverse mistake is subtler: blocking CSS and JavaScript directories prevents Google from rendering pages the way users see them, which degrades how the pages are evaluated even though the HTML itself is crawlable.
Other repeat offenders: blocking a URL and expecting it to leave the index, when crawl-blocking actually preserves whatever the index already holds and hides any noindex you add; writing rules for /Admin/ when the real path is /admin/, which never match because path comparison is case sensitive; putting the file anywhere but the host root, where no crawler will look for it; and adding a bot-specific group without repeating the general rules, silently exempting that bot from everything else.
Before uploading any file, test it. Search Console reports the robots.txt Google last fetched and any parse errors, and its URL inspection shows whether a given page is blocked. Generating from a validated template, then testing the two or three URLs you most care about, catches essentially all of these failures in under five minutes.
Common questions
Robots.txt Generator FAQs
- Where does the robots.txt file go on my website?
- It must be at the root of the host, as in https://example.com/robots.txt, because that is the only location crawlers check. Each subdomain and protocol combination needs its own file, so shop.example.com is not covered by the file on example.com. A file placed in a subdirectory is simply never read.
- How do I block all crawlers with robots.txt?
- Two lines: User-agent: * followed by Disallow: /. This asks every compliant bot to fetch nothing on the site, which is appropriate for staging environments but catastrophic if it reaches production. Note it only blocks crawling; already-indexed URLs can linger in results, and non-compliant bots ignore the file entirely.
- Does robots.txt stop a page from appearing in Google?
- No, it only stops the page being crawled, and Google can still index a blocked URL from external links, showing it without a description. To keep a page out of results, allow crawling and add a noindex robots meta tag or X-Robots-Tag header, or put the content behind authentication. Blocking plus noindex together is self-defeating because the block hides the tag.
- Can I block AI bots like GPTBot with robots.txt?
- Yes, the major AI crawlers, including GPTBot, ClaudeBot, and CCBot, identify themselves and honor robots.txt. Add a group per bot, such as User-agent: GPTBot with Disallow: /, and your search rules remain untouched because each crawler follows only its own best-matching group. Compliance is voluntary, so this is an opt-out signal rather than an enforcement mechanism.
- Does Google support crawl-delay in robots.txt?
- No, Googlebot ignores the Crawl-delay directive entirely and paces itself based on how quickly your server responds. Bing does honor it, expressed in seconds between requests. If Googlebot is genuinely overloading a server, the fix is the crawl rate mechanisms Google provides and faster responses, not a directive it never reads.
- Is robots.txt case sensitive?
- Path matching is case sensitive, so Disallow: /Admin/ does not block /admin/, and the file name itself must be lowercase robots.txt. Directive keywords like User-agent and Disallow are treated case-insensitively by major parsers. When in doubt, write rules to match the exact casing your URLs actually use.
Robots.txt rewards a few minutes of care wildly out of proportion to its size. The whole file is rarely more than twenty lines, yet those lines decide whether crawlers spend their visits on your products and articles or on infinite filter permutations, and one wrong character can quietly take a site out of the running. Generate from intent, test the URLs that matter in Search Console, and the file becomes something you set correctly once and stop thinking about.
ToolDoor's Robots.txt Generator is free and requires no signup. Pick a preset or build your rules visually, add your sitemap line, and download a correctly formatted robots.txt ready to upload to your site root.
Nearby doors