robots.txt for AI: what to allow and what to block

robots.txt has been around since 1994. It was designed for search engines. But now AI crawlers use it too - and your file might be blocking them without you knowing.

The AI crawlers

| Crawler | Company | Powers | |---------|---------|--------| | GPTBot | OpenAI | ChatGPT | | ClaudeBot | Anthropic | Claude | | PerplexityBot | Perplexity | Perplexity AI | | Google-Extended | Google | Gemini, AI Overviews | | meta-externalagent | Meta | Meta AI |

If your robots.txt blocks any of these, that AI assistant cannot read your site and cannot recommend you.

Check your file

Visit yoursite.com/robots.txt and look for:

User-agent: GPTBot
Disallow: /

That blocks GPTBot entirely. A blanket User-agent: * / Disallow: / blocks all bots including AI crawlers.

A good robots.txt

For most sites, this is enough:

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

If you need more control:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

What to still block

Keep sensitive paths private:

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /account/

Sitemap: https://yoursite.com/sitemap.xml

Common mistakes

Security plugins adding broad blocks. WordPress security plugins are the most frequent culprit - they often block unknown bots by default.

Bot whitelists that exclude AI. If you maintain a list of allowed bots, you need to explicitly add the AI crawlers above.

Missing Sitemap directive. The Sitemap: line helps all crawlers - including AI - discover your content.

How to update on your CMS

• WordPress: Yoast SEO > Tools > File Editor, or upload via FTP
• Shopify: Settings > Custom data, or edit robots.txt.liquid in your theme
• Wix: Settings > SEO > robots.txt
• Squarespace: Settings > SEO > robots.txt
• Custom site: Place the file in your web root

Run a free AI Readiness Audit to check if your robots.txt is correctly configured.