Cloudflare's September 15 AI Crawler Deadline, Explained

• From 15 September 2026, Cloudflare's defaults block "mixed-use" AI crawlers from pages that host ads.
• It applies to new customers, new sites added by existing customers, and all existing free-plan customers — not to existing sites on paid plans.
• A mixed-use crawler merges search, training, and agent fetching into a single user agent. The policy is pressure to separate them.
• Pay per crawl turns the binary into a price, using HTTP 402 Payment Required and cryptographically signed crawler identities.
• A default is still only a default. Read your own per-crawler settings — the costly mistake is blocking an answer-indexing bot by accident.

What Cloudflare actually announced

Cloudflare gave AI companies until 15 September 2026 to separate the crawlers they use for search from the ones they use for AI training and agents. Bots that refuse to split risk being blocked by default across a large number of sites.

Specifically, from that date Cloudflare's default settings block mixed-use crawlers from any page that hosts ads. The choice of ad-bearing pages is not arbitrary: it targets exactly the case where a publisher's content funds itself through impressions that an AI answer removes.

The new defaults apply to:

• new Cloudflare customers
• new sites set up by existing customers
• all existing free-plan customers

Existing paid customers keep their current configuration on their existing sites. If you are on a free plan, the change reaches you automatically.

Cloudflare's CEO Matthew Prince framed it as a question of pace rather than principle: "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge."

One supporting figure is worth repeating, because it reframes the cost argument. By Cloudflare's own measurement, more than 50% of AI crawler traffic is spent re-fetching pages that have not changed. That is not a content dispute. That is bandwidth spent transferring identical bytes.

What "mixed-use" means, and why it is the whole point

The word doing the work here is mixed-use, and it is the reason this policy is more interesting than a simple bot block.

AI crawlers do three genuinely different jobs:

Job	What it does	What blocking it costs you
Training	Collects text to train a model	Nothing in search visibility
Answer indexing	Builds the index an assistant searches	Removal from that product's citations
User-triggered fetch	Fetches a page because a user asked about it	That user cannot be shown your page

A mixed-use crawler does more than one of these under a single user agent. That is convenient for the AI company and hostile to the site owner, because it collapses three separate decisions into one. You cannot say "train on nothing, but do cite me in your search product" when both arrive under the same name.

Cloudflare's deadline is leverage aimed at exactly that: separate the bots, and site owners can answer each question independently. Refuse, and the default answer becomes no.

This is the same distinction that decides whether a robots.txt rule helps or hurts you — the full breakdown of which bot does which job is in AI crawlers and robots.txt.

How pay per crawl works

Alongside the blocking policy, Cloudflare has been building a third option between allow and deny: charge.

Pay per crawl is in private beta, and it works over ordinary HTTP semantics rather than a proprietary protocol:

• A crawler requests a paid resource and receives 402 Payment Required, along with a crawler-price header stating the cost.
• A crawler willing to pay resends with crawler-exact-price, or declares a ceiling up front with crawler-max-price.
• On success it gets 200 OK and a crawler-charged header confirming the charge.
• Where charging is configured but the crawler has no billing relationship, the answer is 403 Forbidden.

Identity is the hard part of any scheme like this, since a user agent string is a claim rather than a credential. Cloudflare authenticates crawlers with Ed25519 signatures carried in signature-agent, signature-input, and signature headers, so a bot cannot simply assert someone else's name to get someone else's price.

Publishers configure each crawler as allow, charge, or block, with a flat per-request price across the site. Enforcement happens after existing WAF and bot-management rules, so it layers on top of what you already run rather than replacing it.

Cloudflare has said the marketplace is evolving from pay per crawl into pay per use — charging when content creates value rather than when a bot fetches a file, with Ceramic.ai and You.com as early participants. That is a meaningfully different economic model, and it is early enough that nobody should plan a business around it yet.

What this does and does not change for search

It is worth being precise, because the panic version of this story is wrong.

This does not touch Google Search. Googlebot crawls for Search. Google-Extended is a separate opt-out governing training and grounding for Gemini apps, and it has never been used for crawling, indexing, or ranking in Google Search. Blocking it costs nothing in Search — a point covered in more detail in what AI Overviews are and how they work.

It does potentially touch your AI citations. If a bot that indexes for an answer product gets caught by a default block, you leave that product's index. Given how many AI answers already end without a click, the citation is a large share of the remaining value — see how to get cited in AI search.

It does not require you to do anything if you are on a paid plan with existing sites. It does require you to know that, rather than assume it.

What to check before 15 September

A short, honest checklist. None of it takes long.

1. Find out which Cloudflare plan you are on, and whether your sites predate the change. A free plan or a new site means the new defaults reach you.
2. Open your AI crawler settings and read them. Do not infer them. The question to answer is per-bot: allow, charge, or block.
3. Decide the answer-indexing bots deliberately. OAI-SearchBot, PerplexityBot, and their peers are the ones where a block has a visible cost.
4. Check that your robots.txt agrees with your Cloudflare settings. Two layers with contradictory policies produce behaviour nobody intended.
5. Write down what you decided and when. When traffic from an assistant changes three months from now, you will want to know whether you caused it.

The broader point is the one worth keeping. For two years the only available positions were allow everything and block everything, and most sites picked one by accident. A dated default and a working price mechanism turn that into an actual decision. Make it on purpose.

Sources: Cloudflare — Introducing pay per crawl · Cloudflare — Content Independence Day · TechCrunch — Cloudflare's new policy pushes AI companies to pay for publishers' content · Google Search Central — AI features and your website