Robots.txt Generator

Build a valid robots.txt rule by rule, or start from a preset. Every file is checked against the same rules Google applies as you build it, so what you copy is what crawlers will read. The AI presets carry the current user agents for OpenAI, Google, Anthropic, Perplexity, Apple, Meta, Common Crawl, and ByteDance, which change more often than most robots.txt files do.

Start from a preset

A preset replaces the rules below, so pick one first and then adjust. "Block AI training" leaves you visible in ChatGPT, Claude, and Perplexity; "Block all AI crawlers" does not.

Rules

Each group names one crawler and lists the paths it may and may not fetch. A path is a prefix, so /admin also covers /admin-tools; add a trailing slash when you mean only the folder. Leave a path empty to allow everything.

Sitemap URL

Crawl delay (seconds, ignored by Google)

Your robots.txt

Save the file as robots.txt, upload it to the root of your site so it answers at https://yourdomain.com/robots.txt, then check it with the robots.txt tester. The tester fetches the published file and shows which crawlers your rules actually let through, which is not always what the file appears to say.

Groups, and why the order surprises people

A robots.txt is a set of groups, each opening with one or more User-agent lines. A crawler reads only the most specific group that names it, and falls back to * when no group names it directly. It does not combine them.

That single rule causes most robots.txt mistakes. If your file has a * group disallowing /private and a Googlebot group disallowing /staging, Googlebot is not blocked from /private at all, because it never reads the wildcard group. Anything that must apply to a named crawler has to be repeated inside its group.

The presets above follow this: the AI presets give each crawler its own complete group rather than assuming the wildcard rules carry over.

Blocking a URL is not the same as removing it

A Disallow stops a crawler fetching a URL. It does not remove the URL from the index, and it does not merge a duplicate into its canonical. Both of those need the crawler to read the page first, which is exactly what the block prevents.

So robots.txt is the right tool for pages you do not want fetched at all: an admin area, a cart, a checkout, an internal search that generates a new URL for every query. It is the wrong tool for a page you want out of search results, which needs a noindex tag on a page crawlers may still reach, and the wrong tool for parameter duplicates, which need a canonical tag.

This is why no preset here blocks URLs by their query string. That rule reads as housekeeping and behaves as a site-wide outage on any site whose pagination, filters, or permalinks carry a ?.

Frequently Asked Questions

Where does robots.txt go?

At the root of the host, at https://example.com/robots.txt, and nowhere else. A file at any other path is ignored. Each subdomain needs its own file, so the rules on example.com do not cover shop.example.com.

Does robots.txt keep a page out of Google?

No. It stops a crawler fetching the URL; it does not stop the URL being indexed. A blocked page that other sites link to can still appear in results, listed without a description. To keep a page out of the index, allow crawling and use a noindex meta tag or an X-Robots-Tag header, because Google has to be able to fetch the page to read the directive.

How do I block AI crawlers?

Two presets above cover it, because there are two different decisions here. "Block AI training" names the seven crawlers whose operators say what they fetch may train a model: GPTBot, Google-Extended, ClaudeBot, Applebot-Extended, Meta-ExternalAgent, CCBot, and Bytespider. None of them affect search, so your pages keep ranking and keep being cited in AI answers.

"Block all AI crawlers" adds the eight that fetch pages for answers rather than for training: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Meta-WebIndexer, and Meta-ExternalFetcher. Blocking those removes you from ChatGPT search, Claude, Perplexity, and Meta AI, which is the AI referral traffic a growing number of sites now measure. You should pick this one only if you want no presence there at all.

Two limits are worth knowing. The agents ending in -User or -Fetcher run when a person asks for a specific page, and OpenAI, Perplexity, and Meta all state that robots.txt may not hold for those. And Bytespider publishes no documentation and has been observed crawling paths a robots.txt disallows, so blocking it at your server or CDN is the version that holds.

Why is there no rule blocking URLs with a query string?

Because Disallow: /*?* does more damage than almost any other line people copy into a robots.txt. It blocks every URL carrying a ?, which takes out pagination, filtered category pages, internal search, and every landing page reached with a tracking parameter. On a site still serving /?p=123 permalinks it blocks the site. It also cannot do the job it is usually reached for, since a URL Google may not fetch is a URL whose canonical tag Google may not read, so the duplicates it was meant to consolidate simply go dark instead of consolidating.

If one parameter really is generating an endless crawl space, name that parameter and nothing else, as the Shopify preset does with Disallow: /*?*oseid=. Everything else belongs in a canonical tag.

Should I block Ahrefs and Semrush?

Blocking them hides your backlink profile from competitors researching you, and also from you when you use those same tools on your own site. Most sites leave them allowed, because the competitive intelligence you lose is usually worth more than the intelligence you deny others. The preset exists for the cases where crawler load is a real cost.

Do I need to list my sitemap?

It is not required, and it is worth doing. A Sitemap: line tells any crawler where your sitemap lives without you registering it anywhere, and it works for search engines that have no equivalent of Search Console. It must be a full URL including https.

What about Crawl-delay?

Google ignores it entirely. Bing and Yandex honor it. If Googlebot is crawling harder than your server can handle, the crawl rate setting in Search Console is the mechanism that works. The value you enter is attached to the * group, which is the group every crawler that honors the directive falls back to.