Robots.txt Generator
Build a robots.txt file with rules for search crawlers, AI crawlers, or both. Each directive is explained as you add it, and the tool flags the patterns that most often cause a site to lose traffic by mistake.
- Free, no sign-up
- Runs in your browser
- Nothing uploaded
- Updated Sep 2026
Paths to disallow
One path per line. Paths are case-sensitive and match from the start of the URL.Paths to explicitly allow
Use to carve exceptions out of a broader disallow. More specific rules win.AI crawlers
Training crawlers
Collect content used to train models. Blocking these does not affect search rankings.
Retrieval crawlers
Fetch pages live to answer questions, and cite the source. Blocking these removes you from AI answers.
At a glance
- Location
- Must be at
https://yourdomain.com/robots.txt— the root, nowhere else - Controls
- Crawling, not indexing. A blocked page can still appear in results
- Deindexing
- Use a
noindexmeta tag, never robots.txt - AI crawlers
- GPTBot, ClaudeBot, CCBot, PerplexityBot, Google-Extended and others
- Case
- Paths are case-sensitive; user-agent names are not
- Processing
- Entirely client-side
The distinction that causes the most damage
robots.txt controls crawling. It does not control indexing. Almost every serious robots.txt mistake comes from not understanding that sentence.
If you disallow a URL, a compliant crawler will not fetch it. But if other pages link to
that URL, a search engine can still list it in results — showing the bare URL with no
description, because it was never allowed to read the page. Worse, blocking a page in
robots.txt makes it impossible for the crawler to see a noindex tag on that
page, so a page you are trying to remove stays in the index indefinitely.
The correct approach depends on what you actually want:
- Keep it out of search results: serve
<meta name="robots" content="noindex">and allow crawling so the tag can be read. - Save crawl budget on low-value URLs — faceted filters, internal search, infinite parameter combinations: disallow in robots.txt.
- Protect private content: require authentication. robots.txt is a public file that lists exactly which paths you consider sensitive, which makes it a poor security measure and a useful map for anyone hostile.
AI crawlers and what blocking them costs
A separate set of user-agents now crawls the web for AI systems, and they split into two kinds that are worth treating differently.
Training crawlers — GPTBot, ClaudeBot, CCBot, Google-Extended and similar — collect content used to train models. Blocking them has no effect on your search rankings, because they are distinct from the search crawlers.
Retrieval crawlers — the user-agents that fetch pages live to answer a question someone is asking right now, such as OAI-SearchBot and PerplexityBot. Blocking these removes you from AI-generated answers and the citations that come with them. That is an increasingly real source of referral traffic, and blocking it is a decision rather than a default.
The common middle position is to allow retrieval crawlers, so your content can be cited
and linked, while blocking training crawlers if you object to that use. The generator sets
these up as separate groups so you can choose deliberately. Note also that
Google-Extended controls Gemini training and grounding only — it has no
influence on Google Search ranking whatsoever, despite persistent claims otherwise.
Rules that quietly cost sites traffic
From auditing real files, these are the errors that appear most often:
Disallow: /left in after a launch. This blocks the entire site. It is the single most damaging line in SEO and it usually survives from a staging configuration.- Blocking CSS and JavaScript. Google renders pages to evaluate them. Block the assets and it sees a broken layout, which affects both mobile usability assessment and how content is understood.
Crawl-delay. Google ignores it entirely. Bing honours it, so a delay of 5 means Bing fetches at most about 17,000 pages a day — and on a large site that is a genuine constraint. Use Search Console's crawl rate setting instead if you need to throttle.- Wildcard parameter blocks. Rules like
Disallow: /*?utm_*are broader than they look and can block legitimate URLs. Handle tracking parameters with canonical tags, which is what they are for. - Assuming later rules override earlier ones. They do not. The most
specific matching rule wins, measured by path length, and
AllowbeatsDisallowwhen both match at the same length. - A missing or relative sitemap line. The
Sitemap:directive requires a full absolute URL.
How to use the Robots.txt Generator
-
Enter your domain
Used to build the absolute sitemap URL, which the
Sitemap:directive requires. -
Choose a preset or build rules yourself
The presets cover the common cases. Add specific paths to disallow for anything else — admin areas, internal search, faceted filters.
-
Decide on AI crawlers
Allow retrieval crawlers if you want your content cited in AI answers. Block training crawlers if you object to that use. They are separate choices.
-
Upload and verify
Save the file at your domain root and check it in Google Search Console's robots.txt report, which shows exactly how Google parses it.
Frequently asked questions
Will robots.txt remove a page from Google?
No — and blocking it makes removal harder. A disallowed URL can still be listed, showing the bare URL with no description. To remove a page, allow crawling and serve a noindex meta tag, so the crawler can actually see the instruction.
Does blocking GPTBot hurt my search rankings?
No. GPTBot is OpenAI's training crawler and has no connection to Googlebot or Bingbot. Blocking it affects whether your content is used for training, nothing else. The crawler worth thinking harder about is the retrieval kind, which fetches pages to answer live questions and cites the source.
Should I use Crawl-delay?
Almost never. Google ignores it; Bing honours it and will throttle itself severely, which on a large site means pages go uncrawled for weeks. If you genuinely need to reduce crawl load, use the crawl rate setting in Search Console or return HTTP 429 under pressure.
What is the difference between Disallow and noindex?
Disallow says do not fetch this. noindex says do not list this in results. They solve different problems, and using the first when you meant the second is the most common robots.txt error. Never combine them on the same URL — the crawler cannot read a noindex tag on a page it is forbidden to fetch.Can I have a different robots.txt per subdomain?
Yes, and you need one. Each host is treated separately, so blog.example.com needs its own file at its own root. A robots.txt on the main domain has no authority over a subdomain.
How do I check my file is working?
Google Search Console has a robots.txt report showing the fetched file and how Google parses each rule, and lets you test specific URLs. That is authoritative in a way that reading the file yourself is not, because rule precedence is easy to get wrong.