Robots.txt Generator

Create a properly formatted robots.txt file to control how search engines crawl your website. Add rules for different user-agents, specify allowed and disallowed paths, and include your sitemap.

Crawl Rules

Optional: Include your XML sitemap location

Optional: Delay between crawler requests (not all bots honor this)

# Add rules above to generate robots.txt content

Save this content as robots.txt in your website's root directory.

How to Use This Tool

  1. Add crawl rules - Click "Add Rule" to create rules for different user-agents. The default rule (*) applies to all search engine bots.
  2. Configure directives - For each rule, add Allow or Disallow directives to control which paths can be crawled. Use "/" to affect all paths or specific paths like "/admin/".
  3. Add your sitemap - Enter your XML sitemap URL to help search engines discover all your pages.
  4. Set crawl delay - Optionally specify a delay between requests to reduce server load (note: not all bots honor this).
  5. Copy and deploy - Copy the generated content and save it as robots.txt in your website's root directory (e.g., https://example.com/robots.txt).

Why Robots.txt Matters for SEO

The robots.txt file is a critical part of technical SEO. It tells search engine crawlers which pages they can and cannot access on your website, helping you control your crawl budget and protect sensitive areas.

  • Crawl budget optimization: Block search engines from wasting resources on unimportant pages (admin areas, duplicate content, staging environments).
  • Prevent indexing issues: Stop search engines from crawling pages that shouldn't appear in search results, like login pages or internal search results.
  • Server load management: Use crawl-delay to reduce the impact of crawler traffic on your server.
  • Sitemap discovery: Point crawlers to your XML sitemap so they can efficiently find and index your content.
  • AI bot control: Block AI training bots (GPTBot, CCBot, etc.) if you don't want your content used for AI training.

Common Robots.txt Patterns

Block All Crawlers
User-agent: *
Disallow: /

Blocks all bots from your entire site. Use for staging sites.

Block Admin Area
User-agent: *
Disallow: /admin/
Disallow: /wp-admin/

Prevents indexing of administrative sections.

Allow All Crawling
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Allows full access with sitemap reference.

Block AI Bots
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

Prevents AI training bots from scraping your content.

Frequently Asked Questions

Where should I place my robots.txt file?

The robots.txt file must be placed in the root directory of your website and accessible at https://yourdomain.com/robots.txt. Search engines will only look for it at this exact location - placing it in a subdirectory won't work.

Related Tools

Robots.txt Validator

Check your robots.txt file for errors and test URL blocking rules.

XML Sitemap Validator

Validate your XML sitemap structure and check for common issues.

JSON-LD Schema Generator

Generate structured data markup for rich search results.

Background

Get Google and ChatGPT traffic on autopilot.

Start today and generate your first article within 15 minutes.

Start your free trial
Join with Google
Content Plan