Robots.txt AI Crawler Manager
Decide bot by bot which AI crawlers may access your site, then copy a ready-to-paste robots.txt. Each crawler below is labeled with who runs it and what it is used for.
User-agent list as of July 2026. Crawler names and behavior change; check each vendor's documentation for the latest.
Your robots.txt
The tradeoff, honestly: blocking AI crawlers stops (well-behaved) bots from using your content for model training or answer generation, which protects your content from being repackaged without a visit. But it cuts both ways: assistants like ChatGPT, Claude, and Perplexity increasingly send real referral traffic by citing sources, and if their bots cannot read your site you will not appear in those answers or citations. Many publishers now split the difference: block pure training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider) while allowing user-request fetchers (ChatGPT-User, Claude-User, PerplexityBot) so they stay citable. Also remember robots.txt is voluntary; it does not stop bots that ignore it, and blocking Google-Extended does not affect your normal Google Search ranking. Place the file at your domain root as /robots.txt.