In August 2026, the BBC, New York Times, CNN, Reuters, and USA Today have all blocked Google-Extended, the crawler token Google uses for AI training and grounding. Several publishers are publicly considering blocking Googlebot entirely. ADWEEK framed the split as publishers retreating from AI while creators lean in. This is a real story with real consequences for publishers. But it is also a measurement problem, and measurement problems are Parceit's business.
When major sources block AI crawlers, they change what AI engines can draw from when generating answers. If 30% of authoritative sources become uncrawlable, the answers AI engines produce are built from the remaining 70%. The information landscape shifts. And any tool measuring AI visibility needs to account for that shift, or its numbers are wrong.
This article explains how crawl-blocking affects measurement, what it means for the accuracy of AI answers, and what Parceit does about it.
The measurement problem in plain terms
Imagine you are measuring which restaurants get recommended most often in a city. But half the restaurants just put up curtains so the recommendation engine cannot see inside. The recommendations do not stop. They just shift to whatever restaurants are still visible. Your measurement now reflects a distorted landscape, not the real one.
That is what is happening in AI search. When the New York Times blocks Google-Extended, Google's AI cannot train on or ground responses using New York Times content. When CNN blocks GPTBot, ChatGPT cannot use CNN articles to build answers. The AI does not stop answering questions. It answers them using whatever sources remain available.
This creates two measurement challenges. First, the sources cited in AI answers are not the full set of authoritative sources. They are the subset that allowed crawling. Second, the trend line is moving in one direction. ADWEEK and Digiday both report that more publishers are considering full crawler blocks, not fewer. If that continues, the distortion widens.
What this means for Presence and Portrayal
Parceit's P-Score measures four signal categories that map to the IAB's 4 P's of AI Visibility: Presence, Prominence, Portrayal, and Persuasion. Crawl-blocking affects two of these directly.
Presence asks whether your brand appears in AI responses at all. If you block AI crawlers, your Presence drops to zero for the engines you blocked. There is no partial credit. But there is a less obvious effect: when your competitors block crawlers, your relative Presence can increase even though you changed nothing. You look more visible because others made themselves less visible.
Portrayal asks whether AI describes your brand accurately. When authoritative sources withdraw from the crawlable pool, AI engines fill the gap with whatever remains. That might be a blog post, a forum thread, or a press release. The quality of information about your brand in AI answers can degrade, even though you did not change anything on your site. Your Portrayal score is affected by what other people do with their robots.txt.
Here is the counterintuitive part. Presence and Portrayal can move in opposite directions at the same time. When a source blocks its crawlers, your relative Presence may go up because one fewer site is competing for visibility. You look more visible. But your Portrayal can simultaneously get worse if the departing source had accurate, well-sourced information about your industry. The AI replaces it with whatever remains, which may or may not be as reliable.
There is a flip side. If a site that blocked crawlers happened to be publishing negative or inaccurate information about your brand, its departure could improve your Portrayal. The effect cuts both ways depending on what kind of sources leave the pool. Parceit does not claim crawl-blocking always harms Portrayal. It claims that crawl-blocking changes the information pool, and that change has consequences your measurement tool needs to account for.
Why Parceit's Crawlability score matters more now
Parceit's P-Score includes a Crawlability category worth 25 points. It checks whether your robots.txt blocks critical AI crawlers including GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. It also verifies that your site is actually reachable, that your sitemap is valid, and that your llms.txt file exists for LLM crawlers.
This category has always been important. It is the gate. If crawlers cannot reach your site, nothing else matters. But in the current environment, where publishers are actively debating whether to block crawlers, Crawlability is not just a technical check. It is a strategic decision with measurement consequences.
A brand that blocks GPTBot to make a point about AI fairness has chosen to reduce its own Presence score to zero in ChatGPT. Meanwhile, a competitor that allows all crawlers sees their relative Presence rise. The P-Score reflects this reality accurately, because it starts with a real crawl of your actual robots.txt.
What Parceit does not do
Parceit does not track which third-party sites have blocked crawlers. That is a different kind of measurement problem, and one that several organizations are working on. Parceit measures your site's readiness for AI visibility, not the composition of the crawlable web.
But the connection matters. If your P-Score is high but your brand is still not appearing in AI answers, one possible explanation is that the information ecosystem around your industry has shifted. Competitors or authoritative sources may have withdrawn from the crawlable pool, changing what AI engines can draw from. Understanding crawl-blocking trends helps interpret why a high P-Score might not translate into immediate visibility.
Transparency as methodology
The IAB's framework for AI visibility data introduces a distinction between directional measurement (good enough to spot trends) and decision-grade measurement (rigorous enough to make budget decisions). Parceit's methodology is published and reproducible. The same site produces the same score every time. Every check is a pass or fail with a specific point value.
Being transparent about the limitations of any measurement framework is part of being decision-grade. Crawl-blocking is a real, growing variable that affects the AI information landscape. Tools that ignore it are presenting a cleaner picture than reality supports. Parceit's position is that the P-Score measures your site's readiness accurately, and the broader crawlability trend is context that helps interpret what the score means in practice.
If you want to know where you stand, run a free audit at parceit.com. The Crawlability check takes seconds and tells you immediately whether your site is blocking the crawlers that feed AI answers.