On one side: major publishers like the New York Times, CNN, BBC, Reuters, and USA Today have blocked key AI crawlers from their content. On the other side: individual creators, small businesses, and independent sites opening their doors to every crawler they can find. ADWEEK framed the split in an August 5 article: publishers are retreating from AI while creators are running toward it.
This is not a minor disagreement about robots.txt configuration. It is a structural divergence that will reshape which sources AI engines cite for years. If you run a brand, the outcome of this split determines whether your business appears in AI answers. Here is what is happening and why it matters.
The publisher retreat
ADWEEK reported on August 5 that publishers are retreating from AI crawling while creators lean in. Major publishers including the BBC, New York Times, CNN, Reuters, and USA Today have blocked key AI crawlers such as GPTBot. Digiday reported on August 4 that People Inc.'s CEO said the company was "not turning off Google crawlers yet," suggesting the industry is at a decision inflection point.
The reasons are understandable. Publishers spent decades building audiences and content libraries. AI engines crawl that content for free, use it to generate answers that keep users on the AI platform, and send little or no traffic back. Chartbeat data cited in the IAB's August 3 framework shows search referral traffic down 60% for small publishers over two years, 47% for mid-size, and 22% for large. AI chatbot traffic remains under 1% of total publisher pageviews despite 200%+ growth in 2025. Publishers are watching their traffic evaporate while AI engines profit from their content.
People Inc.'s CEO captured the dilemma in a Digiday interview published August 4. The company is, in the CEO's words, "not turning off Google crawlers yet." That is the publisher prisoner's dilemma: block the crawler and lose traffic, or allow the crawler and watch AI summaries replace your pages.
The creator counter-movement
While publishers block, individual creators and small businesses are deliberately opening up. They are adding GPTBot, ClaudeBot, PerplexityBot, and Google-Extended to their allowed crawlers. Some are creating llms.txt files specifically designed to help LLM crawlers understand their content. They want to be cited in AI answers because a citation in a ChatGPT or Perplexity response can drive highly qualified traffic from users actively making decisions.
The calculus is different for creators. A major publisher loses pageviews when AI summarizes their article instead of linking to it. An independent consultant or small business gains a customer when AI recommends them in an answer. For publishers, AI citation is a substitution risk. For creators and small brands, it is a distribution channel.
This means the content AI engines can draw from is shifting. The biggest, most established sources are withdrawing. Smaller, more willing sources are filling the gap. The answers users get from ChatGPT or Perplexity increasingly reflect whoever is crawlable, regardless of who is most authoritative.
Why this divergence creates winners and losers
When the New York Times blocks GPTBot, ChatGPT stops using New York Times articles to build answers. It does not stop answering questions. It finds other sources. Those sources might be a Substack newsletter, a niche blog, or a brand's own help documentation. The AI builds the best answer it can from whatever is crawlable.
For brands, this is an opening. If your competitors in the space are large publishers who have blocked AI crawlers, and you have not, your content becomes the available source. You get cited. They do not. Not because your content is better or worse, but because yours is accessible and theirs is not.
But this window will not stay open forever. Two things could close it. First, publishers and AI engines could reach licensing deals that bring major sources back into the crawlable pool on commercial terms. OpenAI has already signed deals with several publishers. Second, AI engines could get better at producing quality answers from smaller or less-established sources, which would reduce the advantage of simply being crawlable.
The UK regulatory angle
The UK Competition and Markets Authority has added another variable. In June 2026, the CMA ruled that Google must give publishers functional control to opt out of AI features like AI Overviews and AI Mode, without facing traditional search downranking. Publishers who want to stay in Google Search no longer have to accept AI training as a package deal. The CMA stopped short of ordering a full structural separation of Google's crawlers, but the functional opt-out gives publishers a cleaner choice than they had before.
If publishers can opt out of AI features while keeping their search presence, more of them will. That could accelerate the publisher retreat because the prisoner's dilemma weakens. Publishers could block AI training and grounding while keeping their search referrals. If that happens, the divergence between publishers and creators sharpens further.
What brands should do right now
For most brands, the strategy is clear: allow AI crawlers. The publisher retreat is your opportunity to be cited in AI answers while your larger competitors are not. Blocking GPTBot or Google-Extended makes sense for publishers who sell content directly. It does not make sense for brands who sell products or services and benefit from being recommended.
Three concrete steps. First, check your robots.txt. If GPTBot, ClaudeBot, PerplexityBot, or Google-Extended are blocked, unblock them. Second, add an llms.txt file if you have not already. It helps LLM crawlers understand what your site is about and what content is available for use. Third, run a crawlability audit to verify that your site is actually reachable by AI crawlers. Some sites think they are open but have WAF rules or firewall settings that silently block crawler access.
The divergence between publishers and creators is the most important development in AI search right now. Publishers are protecting content they spent decades building. Creators are racing to fill the gap. Brands that understand this shift and position themselves on the right side of it will be visible in AI answers. Brands that ignore it will not.
Run a free audit at parceit.com to see whether your site is crawlable by every major AI engine. The Crawlability check takes seconds and tells you immediately where you stand.