For years the question of who reads your website was not a question for most companies: search engines were allowed to crawl because they sent visitors back. That exchange shifted with the rise of AI answers. Content is read, summarised and answered — and the click through to the source increasingly fails to happen.
Cloudflare, through whose infrastructure a substantial part of the web runs, has drawn its conclusions. On 1 July 2026 the company rebuilt the control of AI traffic completely — and set a date that becomes a deadline for many websites: 15 September 2026.
1. What actually happens on 15 September 2026
Until mid-2026 Cloudflare essentially offered a single switch: block AI bots — yes or no. That all-or-nothing logic was replaced on 1 July 2026. Since then all customers, including on the free plan can control AI traffic by what a bot actually does on the page.
For each of the three behaviour classes you can set separately: block on all pages, block only on pages with advertising, or do not block at all. What is new above all is that you can separate the useful AI usage from the purely extractive kind.
The real turning point, though, is the change to the default. Cloudflare puts it unambiguously: from 15 September 2026 newly onboarded domains receive changed defaults — bots classified as Training or as Agent are blocked on pages with advertising, while Search remains permitted. From the same date, multi-purpose crawlers that combine search and training in one visit also fall under the new training block. Opting out of the new defaults is possible in the dashboard at any time beforehand.
2. Search, Agent, Training: the three behaviour classes
The real conceptual advance is not the deadline but the categorisation. It is no longer "which provider" that decides, but "which behaviour". Cloudflare describes the three classes like this:
- Search: crawlers that capture and index content in order to answer questions about it later. Here you can expect references to the source or some other fair consideration — so this traffic potentially gives something back.
- Agent: automated behaviour acting in real time on behalf of a specific person. This includes chat fetch bots that retrieve a page during a conversation, and browser agents that click and read for the user.
- Training: crawlers that take content in order to train or fine-tune a model. Here the value flows away entirely: the text ends up in the model, and no return channel to the source exists.
This separation matters because it resolves a debate that was still open in 2026: many providers handled search, answer generation and training through the same crawler — block it and you lost everything; allow it and you released everything. That bundling is exactly what the new rule addresses.
3. Who the change affects — and who it does not
The headlines on this topic are about large publishers. That is understandable but misleading: every website delivered through Cloudflare is affected — and that includes a great many mid-sized sites, often without their owners having consciously decided it. Cloudflare sits inside countless hosting packages, WordPress setups and agency configurations.
Two groups should look closely now:
- Websites with advertising. The new defaults explicitly apply to pages that serve ads. Anyone running display advertising, affiliate banners or ad networks falls within scope — including a small specialist blog.
- Newly created domains and free plans. Anyone onboarding a new domain after 15 September starts automatically on the new default. Check nothing, and you have made a decision without making it.
For a classic company website without ad space, little changes at first. That is no reason to file the topic away, though: the direction is clearly visible, and the controls are now open to everyone — including those who did not know they existed.
4. The conflict of aims: visibility against control
Here lies the strategic core — and the point at which many discussions derail. "Blocking AI crawlers" sounds like protection. In practice it is a trade-off with real costs on both sides.
Block Search and you potentially disappear from the source lists that ChatGPT, Perplexity, Gemini and Copilot show beneath their answers. Those citations are the new visibility channel in 2026. In our article on Generative Engine Optimization we described why presence in AI answers has long stopped being a sideshow for many sectors. Close the door on the search crawlers and you close it on that visibility too.
Leave everything open and you accept that content moves into training data without a reference, a visitor or a payment coming back. For carefully researched content that is a genuine loss of value — particularly in an environment where the share of clickless searches is rising anyway.
5. From pay per crawl to pay per use
The block is only half the story. Back in 2025 Cloudflare had presented Pay per Crawl, a model in which AI providers pay for access to content — site owners set the prices, AI providers decide whether to access. Numerous large media houses joined at the time.
In 2026 Cloudflare developed the model further: under the heading Pay per Use, what gets paid for is no longer the mere retrieval of a page but the actual use — the case where content feeds into an AI answer or an agent accesses it for a specific task. Economically that is the more logical approach, because it measures value where value arises.
For mid-sized companies this is a signal rather than a revenue stream for now: such models pay off first for websites with large content holdings. The direction is remarkable all the same — free access to web content, taken for granted for 25 years, is becoming a negotiable service.
6. What happens if you block training?
The honest answer: in most cases less than you fear — but also less than you hope.
Your AI visibility is usually preserved
As long as Search remains permitted, AI systems can continue to index your page and cite it in answers. That is precisely the point of the new separation. Your chance of appearing in an answer to "who offers X in this city?" still depends on content, structure and authority — not on training permission. How to strengthen those factors is broken down in our article on E-E-A-T and AI authority.
Your content does not retroactively vanish from the models
What has already flowed into training data cannot be recalled by a block. A block only works forwards. Anyone expecting their brand to disappear from existing models will be disappointed — and that is usually not the intention anyway.
Not every crawler obeys the rules
A block at CDN level is considerably more binding than an entry in robots.txt, but it is not absolute protection either. Crawlers that present themselves as ordinary browsers can only be identified with considerable effort. A realistic expectation: you are steering the reputable field, not the entire internet.
7. Six steps to the right setting
Concrete, and in this order — the effort is about half an hour:
- Check whether Cloudflare is involved at all. Many owners do not know. A look at the domain's DNS settings, or a question to the host or agency, settles it in minutes.
- Inventory your ad space. Does your website serve ads — including affiliate banners or embedded networks? Then the new defaults apply to you directly.
- Decide the three classes deliberately. For most mid-sized websites: allow Search (visibility), allow Agent (your customers increasingly use assistants), weigh Training consciously.
- Set it before 15 September. Anyone who wants to keep their current settings can opt out of the new defaults in the dashboard beforehand. After that it is a correction rather than a decision.
- Measure the effect. After the change, watch whether referral traffic from AI systems drops. How to measure that channel at all is shown in our article on the AI assistant channel group in GA4.
- Document the decision. Write down what you set and why. In twelve months nobody will remember whether the configuration was intent or default.
8. Without Cloudflare: robots.txt and its limits
If your website does not run through Cloudflare, nothing changes automatically on 15 September. You can still steer — only with weaker means.
The obvious tool is robots.txt. Individual AI crawlers can be addressed there: GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot, Google-Extended, Bytespider and others. The decisive difference from a CDN block: robots.txt is a request, not a technical barrier. Reputable providers observe it, others ignore it.
A common and expensive mistake at this point is blocking everything called "bot" across the board. In doing so you may also shut out the crawlers that would have cited your content in AI answers — a self-inflicted loss of visibility. Anyone who wants to block bindingly needs a rule at server or firewall level anyway.
And one more clarification, because the question comes up regularly: an llms.txt is not an access control instrument. It describes content; it permits and forbids nothing.
9. Conclusion: a decision you should make actively
Cloudflare's new rules are less a technical update than an economic turning point. They make visible that access to web content is not a law of nature but an exchange — and that this exchange has fallen out of balance in recent years.
For mid-sized companies a blanket block is rarely the right answer. Anyone who wants to appear in AI answers has to remain findable. The sensible stance is more differentiated: keep search open, allow agents, decide on training deliberately — and do not leave that decision to the accident of a default.
Also worth reading: Is your website visible in ChatGPT? and Ads in AI answers: what advertising in ChatGPT and Perplexity means for smaller companies.
10. FAQ: the key questions about AI crawlers
What changes at Cloudflare on 15 September 2026?
From 15 September 2026 new defaults for AI traffic apply to domains newly onboarded at Cloudflare: crawlers classified as Training or Agent are blocked on pages with advertising. Crawlers with Search behaviour remain permitted. Multi-purpose crawlers combining search and training fall under the new training block from that date. Anyone wanting to keep the old settings can opt out in the Cloudflare dashboard at any time before 15 September.
What is the difference between Search, Agent and Training?
Cloudflare no longer distinguishes AI traffic by provider but by behaviour. Search means crawlers that index content in order to answer questions about it later — here you can expect references or other consideration. Agent means automated access acting in real time on behalf of a person, such as chat fetch bots or browser agents. Training means crawlers that take content to train or fine-tune a model. Each of the three classes can be set separately to allow, block everywhere, or block only on pages with advertising.
Does blocking AI crawlers hurt my visibility in ChatGPT?
It depends which class you block. Blocking Search risks disappearing from the source lists of AI answers — that hits AI visibility directly. Blocking only Training prevents the content being used for model training but generally leaves you citable. For most mid-sized companies that separation is the sensible setting: keep Search open, decide Training deliberately.
What applies if my website does not run through Cloudflare?
Then nothing changes automatically for you on 15 September 2026. Control is still possible, but with weaker means: robots.txt lets you address individual AI crawlers such as GPTBot, ClaudeBot, PerplexityBot or Google-Extended. That is only a request, though, not a technical block — reputable providers observe it, others do not. Anyone who wants to block bindingly needs a block at server or CDN level.
