robots.txt Generator — Build Yours Without the Gotchas
Presets, path rules and AI-crawler opt-outs — with the syntax mistakes designed out.
🔒 Runs in your browser — files never uploaded ⚡ No signup 💯 Free
Ticked crawlers get their own Disallow: / block. Search indexing
(Googlebot, Bingbot) is unaffected.
Build a correct robots.txt: start from a preset, list the paths to block, tick the AI training crawlers you want opted out, add your sitemap — and copy a file with the classic syntax mistakes designed out (paths are normalised to start with /, because a Disallow without it silently matches nothing).
The AI-crawler section reflects the new reality: GPTBot, Google-Extended, CCBot and friends respect robots.txt, and blocking them does not affect your search rankings — training crawlers and search crawlers are separate opt-outs.
How to use robots.txt Generator
- Start from a preset — allow everything, WordPress, or block-all for staging.
- List paths to block, one per line: admin areas, carts, internal search results.
- Tick AI crawlers to opt out of, if that's your policy. Search bots are untouched.
- Add your sitemap URL and download robots.txt for your site root.
What robots.txt is — and pointedly is not
It's a politeness protocol: well-behaved crawlers read it and comply. It is not security — blocked URLs are still reachable by anyone, and the file itself is public (yours, Google's, everyone's — try /robots.txt on any site). Never "protect" an admin area by listing it here; that's a signpost, not a lock. Access control belongs in the server.
The mistakes this generator prevents
- Missing leading slash —
Disallow: adminmatches nothing;Disallow: /adminis what you meant. - Blocking everything by accident — a lone
Disallow: /deindexes the site. Fine for staging (that's the preset), catastrophic in production; the site's own robots.txt here serves exactly that on non-production environments for this reason. - Blocking CSS/JS — Google renders pages; blocking assets makes it see broken layouts and can hurt rankings. None of the presets do it.
On blocking AI crawlers
The listed bots collect training data, identify themselves honestly, and respect robots.txt. Blocking them is a legitimate policy choice with no search-ranking cost — Googlebot and Google-Extended are deliberately separate tokens. The honest caveat: robots.txt only stops crawlers that choose to obey it.
After you deploy
The file must live at exactly /robots.txt in the site root. Verify what a crawler sees with the header checker, and confirm your sitemap line points at a URL that answers 200.
Frequently asked questions
Does robots.txt hide pages from people?
No — it's a request to crawlers, and the file itself is public. Anyone can read yours and visit every path you listed. For actual privacy you need authentication; for keeping a page out of the index while still crawlable, you want a noindex meta tag instead.
Will blocking GPTBot or CCBot hurt my Google rankings?
No. Search crawling (Googlebot, Bingbot) and AI-training crawling are separate user-agents — Google deliberately split Google-Extended from Googlebot so sites could opt out of training without touching search. Block the training bots and your rankings are unaffected.
Where do I put the file?
At the site root, exactly /robots.txt — crawlers look nowhere else. One file per host: subdomains need their own, and it must be reachable over both the www and bare domain if both serve your site.
What does "Disallow:" with nothing after it mean?
Allow everything — it's the standard explicit way to say "no restrictions". An empty file works too, but the explicit form documents intent and survives copy-paste editing better.
Should I block my internal search results pages?
Usually yes (/?s=, /search/) — they generate infinite thin URLs that waste crawl budget, which is why the WordPress preset includes them. Your real content pages should never be blocked.