Skip to main content

Robots.txt Generator

v1.0.0

Generate a robots.txt file to control crawler access to your website.

Generates and validates robots.txt files with a visual rule builder for allowing, disallowing, and setting crawl-delay.

How to use
  • Add rules using the + buttons for each user-agent/path combination.
  • Enable sitemap reference to include your sitemap URL in the output.
  • Validate an existing robots.txt by pasting it into the Validate tab.
  • Try a preset from the example picker to see sample input.
User-agent:
disallow:
allow: