Robots.txt Tester and Validator

Enter a domain and the tester fetches its live robots.txt, validates every directive, and shows exactly which crawlers may reach which URLs. Google's own tester was retired in 2023, so this applies the matching rules Google documents: longest pattern wins, and Allow beats Disallow on a tie.

robots.txt contents

Test URLs against this file

Enter one path or full URL per line to see whether the rules above allow it. A path that comes back blocked is one crawlers never fetch, so if a page you want ranked appears here as blocked, the rule named beside it is the line to change.

Paths or URLs

Crawler

Crawler coverage

Which search and AI crawlers this file lets through, checked against the first path above.

Reading a robots.txt file

A robots.txt file is a set of groups. Each group opens with one or more User-agent lines naming the crawlers it applies to, and continues with the Allow and Disallow rules those crawlers should follow. A crawler reads only the most specific group that names it, and falls back to the * group when no group names it directly.

That last point causes most of the surprises. If your file has a User-agent: * group and a User-agent: Googlebot group, Googlebot ignores the wildcard group entirely, including its Sitemap-adjacent rules and any disallow you assumed applied to everyone. The coverage table above resolves this for you and names the group each verdict came from.

The file also has to load. A crawler that asks for /robots.txt and gets a 404 concludes the site has no rules and fetches whatever it can reach. A crawler that gets a 500 concludes nothing at all, and RFC 9309 tells it to assume a full disallow until the file comes back. Google follows that, and reads a 429 as the same kind of failure. So the most damaging thing a robots.txt can do is fail to load: a site whose server errors on that one address drops out of crawling while every page on it still works. This tester reports the two separately, and shows every crawler as blocked when the file is unreachable.

Paths support two wildcards. An asterisk matches any run of characters, so Disallow: /*? blocks every URL with a query string. A dollar sign anchors the match to the end of the path, so Disallow: /*.pdf$ blocks PDFs but not /file.pdf.html. Everything else is a literal prefix match starting at the root of the path.

The five mistakes worth checking for

Disallow: / left over from staging. A site launches with the file that blocked the staging server, and the traffic never arrives. The validator flags this as an error rather than a warning, because it is almost never intentional on a live site.
Blocking CSS or JavaScript. Google renders pages before it judges them. A rule that blocks /assets or /wp-content can leave Google rendering a page that looks broken, and a page that looks broken ranks like one.
Using robots.txt to hide a page. Blocking a URL prevents crawling, not indexing. If the page must stay out of the index, it has to be crawlable so the noindex can be read.
A Noindex directive in robots.txt. Google stopped supporting it in 2019 and ignores it silently, so a file that relies on it is leaving pages indexed while appearing to handle them.
A robots.txt that does not load. A 404 costs little, because crawlers then treat the site as having no rules. A 5xx costs the whole site: Google assumes a full disallow for as long as robots.txt returns a server error, so a file behind a flaky server takes pages out of crawling without a single rule changing.

Checking many sites

If you manage SEO for a portfolio of sites, the robots.txt is worth checking on every handover and after every replatform, because it is the one file that can remove a whole site from search in a single line. The URL of this tool accepts a ?url= parameter, as in ?url=example.com, which fetches and validates that domain the moment the page opens. Keep one bookmark per client and open them in sequence.

RankYak does the same check continuously on the sites it runs, alongside the rest of the technical audit, and writes and publishes the articles those sites rank with. If you would like the checking to happen without you remembering to open a tool, that is what the product does.

Frequently Asked Questions

How do I test my robots.txt file?

Enter your domain above and the tester fetches the live robots.txt from /robots.txt, validates every directive, and shows which crawlers may reach which paths. You may also paste a file you have not published yet by switching to the paste tab.

Google removed its robots.txt tester. What replaced it?

The old tester inside Search Console was retired in 2023. Search Console now shows the robots.txt report under Settings, which lists the files Google fetched and when, but it no longer lets you test a URL against a file you are editing. This tool covers that gap: it applies the same matching rules Google documents, so you may check a rule before you publish it.

Which rule wins when Allow and Disallow both match?

Google resolves the conflict by the length of the matching path pattern: the longest pattern wins, and when two patterns are the same length, Allow wins. So Allow: /blog/public beats Disallow: /blog for the URL /blog/public/post. This tester follows the same rule, and names the winning line for every result.

Does robots.txt stop a page from appearing in Google?

No, and this is the most expensive misunderstanding in technical SEO. Robots.txt stops a crawler from fetching a URL; it does not stop Google from indexing it. A blocked URL that other pages link to may still appear in results, listed without a description. To keep a page out of the index, allow crawling and use a noindex meta tag or an X-Robots-Tag header instead.

How do I block AI crawlers such as GPTBot or ClaudeBot?

Name each crawler in its own group and disallow everything, for example User-agent: GPTBot followed by Disallow: /. The coverage table above checks the AI crawlers alongside the search engines, so you may confirm your rules landed. Note that Google-Extended controls Gemini training but does not affect Google Search, and that blocking a crawler is a request the crawler chooses to honor.

Where does robots.txt have to live?

At the root of the host, at https://example.com/robots.txt, and nowhere else. A file at /blog/robots.txt is ignored. Each subdomain needs its own file, so shop.example.com is not covered by the rules on example.com, and http and https are treated as the same host by Google but not by every crawler.

What happens if robots.txt returns a server error?

Crawlers read the two failures in opposite directions. A 404, or any other ordinary 4xx, means the site publishes no rules, so everything is crawled. A 5xx means the crawler could not learn the rules at all, and RFC 9309 has it assume the strictest ones instead: Google treats a robots.txt that keeps returning a server error as a full Disallow: / and stops crawling the site until the file is served again. Google reads 429 the same way it reads a 5xx. A broken server at that one address therefore costs far more than a missing file does, and this tester names which of the two you are looking at.

Why does my robots.txt still show the old rules?

Google caches robots.txt for up to 24 hours, so a change you published this morning may not apply yet. This tester fetches the file live, without a cache, so what you see here is what is on the server right now. If the two disagree, the server is correct and Google will catch up.

Is Crawl-delay supported?

Google ignores Crawl-delay entirely. Bing and Yandex honor it. If Googlebot is crawling your site harder than it can handle, the crawl rate setting in Search Console is the mechanism that works.