Robots.txt Generator
Build a valid robots.txt for your website. Wizard or full controls, smart SEO score, undo/redo, import existing files, JSON backup and one-click download β all in your browser.
Smart Setup Wizard
Site & Directives
How to Use the Robots.txt Generator
- Try Easy Mode first β answer 4 quick questions and we generate a safe config automatically.
- Or switch to Advanced Mode to control every group, rule, crawl-delay and global directive.
- Watch the Smart SEO Score β it updates live and flags dangerous configs (blocking Google, blocking assets, conflicts).
- Import an existing robots.txt β paste it in via the Import button and it's parsed into editable groups.
- Use presets for quick starts, and undo/redo freely while experimenting.
- Generate, review the output, then Copy, Print, Download, or Export as JSON for backup.
Easy Mode vs Advanced Mode
Easy Mode is a beginner-friendly wizard. It asks about your platform, whether Google should crawl, and whether AI crawlers should be blocked β then generates a safe, sensible configuration. Advanced Mode exposes the full editor for every group and directive. Your choice is remembered on this browser.
How robots.txt works
The file lives at the root of your site: https://example.com/robots.txt. Crawlers fetch it before crawling and follow the rules for their user-agent. A * group applies to everyone. Allow rules always take precedence over Disallow rules at the same specificity β and Google uses the longest match wins rule.
Google-Extended is not Googlebot
Googlebot controls Google Search indexing. Google-Extended controls whether your content trains Gemini models. Blocking Google-Extended has no effect on search indexing. Blocking Googlebot removes your site from Google Search entirely. This tool treats them separately in the AI crawler preset.
Allow vs Disallow
Disallow tells crawlers to stay out of a path. Allow explicitly lets them back in β usually paired with a broader Disallow to carve out an exception. For example, block /admin/ but allow /admin/public/.
Wildcards
Google supports * (any sequence) and $ (end of URL). So Disallow: /*.pdf$ blocks all PDFs, and Disallow: /*?sort= blocks sorted URLs. Use sparingly β an over-broad pattern can deindex valuable pages.
Crawl-delay
This is not part of the official robots.txt spec. Google ignores it entirely; Bing, Yandex and others may respect it. If you need rate limiting, use it sparingly or rely on your CDN and Search Console settings instead.
Is my data uploaded?
No. Everything runs in your browser. Settings are auto-saved to this browser's localStorage only β nothing is sent to a server.
Frequently Asked Questions
noindex meta tags or Search Console removal tools.Google-Extended is added (AI training opt-out)./wp-includes/ and /wp-content/plugins/, which is standard for crawl-budget reasons. If your pages depend on assets from those paths, Googlebot may not render them. If SEO is critical, whitelist the exact CSS/JS files you need with Allow: lines.