robots.txt and AI crawlers: are you blocking ChatGPT by accident?

One of the first things we check for a new customer is robots.txt. Surprisingly often, it contains a rule that blocks GPTBot or another AI crawler. The result: that AI model can't read your site, doesn't cite you and rarely names you. This article explains how to check and fix it.

Which crawlers exist

• GPTBot — OpenAI, used for ChatGPT and its search feature.
• OAI-SearchBot — OpenAI, specifically for search results in ChatGPT.
• PerplexityBot — Perplexity.
• Google-Extended — Google, for Gemini and AI features (separate from the regular Googlebot).
• ClaudeBot — Anthropic, for Claude.
• Applebot-Extended, Bytespider, CCBot — other crawlers you may come across.

How to check

Open yourdomain.com/robots.txt in your browser. Look for rules like 'User-agent: GPTBot' followed by 'Disallow: /'. Also look for a general 'User-agent: *' with 'Disallow: /', which blocks everything. Then check your CDN or firewall (Cloudflare, for example, has a toggle to block AI bots) and any security plugins in WordPress.

Note: allowing the regular Googlebot isn't enough. Google-Extended is a separate user agent, and blocking it limits your visibility in Gemini and possibly in Google's AI features.

The trade-off: training versus citing

Some business owners block AI crawlers on purpose, because they don't want their content used to train models. That's a legitimate choice, but know the price: the same crawler traffic is used to give current answers and cite sources. For most small businesses, getting mentioned outweighs the theoretical training risk of a website full of public business information.

What to put in robots.txt

A simple, safe setup: explicitly allow the crawlers above for your public pages and exclude only what's truly private (customer portals, search results, shopping carts). Add a sitemap line as well. Check again after every website change. We see robots.txt files revert after a theme update or migration.

Monitor instead of checking once

Because this breaks so easily without anyone noticing, we check crawlability every night and keep the history. A block that appears on Tuesday becomes an action on Wednesday, not a mystery in your numbers three months later.