AI Crawlers and robots.txt: GPTBot, ClaudeBot, and the Rest

What are the three kinds of AI crawler?

Role	What it does	Agents
Training	Collects text to train or fine-tune foundation models	GPTBot, ClaudeBot, anthropic-ai, CCBot, Google-Extended, Applebot-Extended, meta-externalagent, Bytespider
Answer index	Continuously indexes pages so the product can answer live and cite sources	OAI-SearchBot, Claude-SearchBot, PerplexityBot
User-triggered	Fetches one page on demand because a user pasted or asked about it	ChatGPT-User, Claude-User, Perplexity-User

The distinction that matters most: GPTBot and OAI-SearchBot are both OpenAI, and blocking one does nothing to the other. A robots.txt rule that disallows GPTBot limits training use while leaving ChatGPT search indexing untouched. A rule that disallows OAI-SearchBot removes you from ChatGPT's search citations while training crawls continue under whatever GPTBot permits. Plenty of sites have accidentally chosen exactly the opposite of what they intended.

The -Extended tokens deserve their own note because they are so often misunderstood. Google-Extended does not crawl anything. It is a permission token: it governs whether content already fetched by Googlebot may be used for training and grounding in Gemini apps. It has no effect on Google Search indexing, ranking, or AI Overviews. Applebot-Extended works the same way relative to Applebot and Siri. Both are close to free opt-outs if training use is your concern — which is precisely why they are the first thing to reach for.

What does the crawl actually cost you?

This is the part that changes people's minds, and it is worth having the numbers before you decide.

Cloudflare publishes crawl-to-refer ratios — pages crawled per referral visit sent back to the site. The figures move around a lot between reporting periods, so treat them as orders of magnitude rather than precise values, but the shape is consistent:

Crawler	Approximate crawl-to-refer ratio
Googlebot	~5:1
DuckDuckGo's DuckAssistBot	~1.5:1
OpenAI's GPTBot	roughly 900:1 to 1,300:1
Anthropic's ClaudeBot	roughly 10,000:1 to 24,000:1

Traditional search is a trade: the crawler takes your content and sends readers back. A training crawler with no consumer search product on the other end sends essentially nothing, because there is no product surface that links out. That is not an accusation of bad faith — it is a structural fact about what the crawler is for, and it is the honest basis on which to decide whether you want to participate.

By Cloudflare's May 2026 measurements, AI crawlers accounted for roughly 20% of verified bot traffic, with AI-search bots adding several points more. This is now a real share of your server load.

What should your robots.txt say?

There is no universally correct policy. There is a correct way to decide: separate the two questions.

Question one — do you want to be found and cited in AI products? If yes, allow the answer-index and user-triggered agents. Blocking OAI-SearchBot or PerplexityBot guarantees you will not be cited there, and no amount of content optimisation overrides that.

Question two — do you want your text used for training? Independent of the first. If your content is the product you sell, or you have licensing arrangements to protect, blocking training crawlers is defensible and increasingly common.

The default we run on this site is allow everything, block the app surface. The reasoning is written into the config: assistant citations are a real referral channel for a publication, and the content is free to read anyway. The rules are listed as an explicit group rather than left to the wildcard, so the policy is a visible decision rather than an accident:

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: PerplexityBot
Allow: /
Disallow: /dashboard
Disallow: /login
Disallow: /api/

A publisher who sells access would reasonably invert the training half:

# Allow the answer index, refuse training
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Two practical notes when you write these:

• Group tokens explicitly. Most implementations apply only the most specific matching group, so an agent named in its own group ignores the * group entirely — including any Disallow lines you assumed were global. If you name an agent, restate its full policy.
• Do not forget the useful exceptions. A blanket Disallow: /api/ will silently break link previews if your Open Graph image is generated by an API route. A longer, more specific Allow beats a shorter Disallow in every major implementation, so carve the exception back out explicitly.

Does any of this actually stop anyone?

Only partly, and you should be clear-eyed about it. robots.txt is a convention, not an access control. It has no enforcement mechanism; a crawler obeys it because its operator chose to.

The major operators document compliance, and Anthropic publicly commits to honouring robots.txt across its crawlers and not bypassing access controls. Others have a worse record: Bytespider and some Perplexity fetchers have both been documented disregarding it.

If you need enforcement rather than a request, you need the network layer:

• WAF or CDN rules matching user agent and verified source IP ranges. Most major crawlers publish their IP ranges precisely so you can verify them; user-agent strings alone are trivially spoofed.
• Cloudflare's bot controls, which now include managed AI-crawler blocking and pay-per-crawl options.
• Authentication. Content behind a login is not crawled by anything that respects the boundary, which is most things.

Use robots.txt to state policy clearly and to steer the well-behaved majority. Use the network layer when the answer has to be enforced.

A checklist

1. Read your current robots.txt and check whether you are blocking GPTBot while allowing OAI-SearchBot, or the reverse. Confirm that is what you meant.
2. Decide the training question separately from the visibility question. They are not the same decision.
3. Add Google-Extended and Applebot-Extended to your disallow list if training use concerns you at all — they cost nothing in Search or Siri.
4. Check your logs for AI crawler volume. If it is meaningful load, rate-limit at the edge rather than hoping a text file solves it.
5. Do not expect a hint file to help. llms.txt is not a crawler directive and, on the evidence, is barely read — see does llms.txt do anything.
6. Remember what blocking costs. If you are not in the answer index, you cannot be cited, no matter how good the page is. How AI Overviews work covers the retrieval side of that trade.

Sources: Google Search Central — AI features and your website · Cloudflare Blog — The crawl before the fall of referrals · Cloudflare Blog — A deeper look at AI crawlers · OpenAI — GPTBot documentation · Anthropic — Supporting responsible web crawling