ROBOTS.TXT GUIDE
Block GPTBot without blocking Google
A robots.txt file can set different rules for different crawlers. The safe pattern is to target the AI crawler you want to restrict, then leave your public site available to ordinary search crawlers.
Use a dedicated user-agent rule
Put a GPTBot rule in its own group and use Disallow: / when you do not want that crawler to access the site. Do not put Googlebot in the same group or use a broad rule by accident.
Keep public pages available to search
Your general User-agent: * group should continue to allow the pages you want indexed. Test the final file for overlapping rules before publishing it at your domain root.
- Review every Disallow line for unintended directories.
- Keep your public XML sitemap accessible.
- Validate important URLs in Google Search Console and Bing Webmaster Tools.
Know the limit of robots.txt
Robots.txt is a request to compliant crawlers, not authentication or copyright enforcement. Use access controls for private material and noindex for crawlable pages that should not appear in search results.
Ready to generate your rules?
Use the generator to create a readable, downloadable robots.txt, then publish it at your domain root.
Create robots.txt