HELP CENTER -> SEO & AEO
Let ChatGPT, Claude and other AI crawlers read your site
What your robots.txt allows
- Every search engine and AI crawler can read your published pages.
- It names search engines such as Googlebot, Bingbot and Applebot, and AI crawlers such as GPTBot, ChatGPT-User, PerplexityBot, Claude-Web, anthropic-ai, Google-Extended and CCBot.
- Crawlers it doesn't name, such as ClaudeBot, are allowed too.
- It keeps crawlers out of /admin/, /api/ and /dashboard/.
Check it
- Open yourdomain.com/robots.txt.
- To add your sitemap to it, turn on Auto-generate on publish in settings under SEO.
Limits
- You can't block a single crawler or add your own rules.
- To block every crawler, set a password in settings, under General and Site Access.
- Your free preview address always blocks crawlers. Use your own domain to be found.
Common questions
Why does my robots.txt say "Disallow: /" for everyone?
Either a site password is set, or you opened your free preview address.
Does this help me show up in AI answers?
It lets AI tools read your pages, which they need before they can cite them. Clear, direct answers on your pages help too.
On this page
What your robots.txt allowsCheck itLimitsCommon questions
Prerequisites
Still have questions?
Ask Pixel in your dashboard. It can make many changes for you, or take you to the right screen. Our team can also help you plan, build or move your site.
