The old way of blocking AI bots is gone. Cloudflare replaced the single toggle with a three-category system, and the new defaults take effect September 15. If you have not looked at your settings since the change, your site could quietly lose visibility across Google AI Overviews, ChatGPT, and Perplexity without any warning.
Here is what changed, what the deadline means, and how to check whether your site is at risk before it is too late.
What Cloudflare changed
For years, Cloudflare offered a simple switch: block AI bots or allow them. That switch is gone. Cloudflare now categorizes AI crawlers into three tiers based on what they do, not who they are.
The first tier is Search. These crawlers retrieve content to generate answers in real time. Google AI Overviews, Perplexity, and ChatGPT Search all rely on Search-tier retrieval. When a user asks a question, the AI fetches relevant pages and synthesizes an answer from what it finds.
The second tier is Agent. These crawlers perform actions on behalf of users. Think of AI shopping assistants that navigate a site, compare products, or complete a checkout. The Ninth Circuit ruled in August 2026 that when an AI agent acts on behalf of a user, the user, not the AI company, is the one accessing the site under federal computer fraud law. The court noted its holding is narrow and limited to the specific technology before it. This legal precedent means agentic crawlers are here to stay and will only grow.
The third tier is Training. These crawlers collect content to build or improve AI models. This is the tier most publishers want to restrict, because training use feels like uncompensated extraction.
The reason this matters is that these categories overlap. A single crawler can serve multiple purposes. And that overlap is where the danger sits.
The September 15 default change
On September 15, Cloudflare's new defaults take effect for new customers and new sites. Training and Agent crawlers will be blocked by default on pages that display ads. The intention is reasonable: publishers should not have their content scraped for model training or agentic actions without consent.
But Search Engine Journal reported a critical problem. The new classification system can accidentally block Googlebot. When a site blocks Training crawlers, the same rules can catch Googlebot because the classification boundaries are not clean. Googlebot is primarily a Search-tier crawler, but its behavior patterns overlap with Training crawlers in ways that Cloudflare's classification does not always separate correctly.
If Googlebot gets caught in a Training block, your site does not just lose AI visibility. It loses traditional search visibility too.
Patreon already made the jump
Patreon abandoned robots.txt entirely and now relies on Cloudflare to enforce crawler rules. This is a signal of where things are heading. Robots.txt works on the honor system. Crawlers can ignore it. Cloudflare enforcement actually blocks them at the network level. More sites will follow Patreon's lead as the limitations of robots.txt become clearer.
But moving from robots.txt to Cloudflare without understanding the three-tier system is trading one problem for another. With robots.txt, the worst case is that a crawler ignores your rules. With Cloudflare, the worst case is that you accidentally block the crawlers you actually want.
How to check your site before September 15
First, find out whether you are using Cloudflare's AI crawler controls. If your site is behind Cloudflare and you have not changed any AI bot settings since the overhaul, you are on whatever the default configuration is. That default is about to change.
Second, check which crawlers are currently reaching your site. Before September 15, check your server logs to confirm which AI crawlers are visiting your site. You want to see Googlebot, GPTBot, PerplexityBot, and ClaudeBot. After September 15, check again. If any of those crawlers stop appearing, Cloudflare's new classification system may have sorted them into the wrong tier and blocked them.
Third, test the overlap. If you decide to block Training crawlers, check whether Search crawlers are also affected. The safest approach is to block Training and Agent tiers selectively, page by page, rather than applying a blanket block that might catch Googlebot.
Fourth, if you are a new Cloudflare customer or are adding new sites after September 15, review the default settings immediately. Do not assume the defaults match your intentions.
Why this matters beyond Cloudflare
Cloudflare is expanding into AI visibility and governance as a business line, as reported by Simply Wall St on August 7. The company that controls your crawler access is also positioning itself as the company that measures your AI visibility. That is a conflict of interest worth paying attention to. The same vendor that decides which crawlers can reach your site should not be the only source of truth on whether your site is visible in AI answers.
This is why independent measurement matters. The IAB published its visibility framework on August 3, defining four dimensions: Presence, Prominence, Portrayal, and Persuasion. Presence starts with crawlability. If your site is blocking the crawlers that feed AI answers, your Presence score is zero. But you cannot trust the company that built the blocking mechanism to also tell you accurately whether the blocking is hurting you.
What to do today
If September 15 is approaching and you have not reviewed your Cloudflare settings, do it now. The deadline is not a marketing event. It is a configuration change that can silently remove your site from AI search results.
Run a free audit at parceit.com. The audit checks whether AI crawlers can actually reach your site, across every major AI search engine. If Cloudflare's new defaults are blocking the crawlers you need, the audit will show you exactly where the problem is and what to fix.
Do not wait for September 15 to find out your site disappeared.