Block GPTBot without blocking Google
Add a focused GPTBot rule while keeping Googlebot and Bingbot able to crawl public pages.
Read guide →SEARCH VISIBILITY GUIDES
AI crawlers, robots.txt, and search visibility
Clear, practical guidance for publishing crawler rules without accidentally making your public site harder to find in Google or Bing.
Google-Extended and Google Search are separate controls
Understand why a Google-Extended preference is not the same as blocking Googlebot from Search.
Read guide →Robots.txt vs noindex: choose the right control
Use crawler rules, noindex, and authentication for the distinct problems they are designed to solve.
Read guide →XML sitemaps for public sites
Build a useful XML sitemap that lists the canonical public URLs you want search engines to discover.
Read guide →