We fetched and analyzed the robots.txt files from the top 1,000 websites on the internet. The goal was simple: find out how many sites are blocking AI search crawlers, which bots get blocked most often, and what that means for the future of AI search visibility.
The results surprised us. One in four sites with a robots.txt file is actively blocking at least one critical AI search crawler. Some of the biggest names on the internet are blocking nearly all of them.
This is original research. Every number in this article comes from real data we collected on July 31, 2026. You can read the full methodology at the end.
What we did
We downloaded the Tranco top-1 million domain list, which ranks websites by a combination of Google and Alexa-style metrics. We took the top 1,000 domains from that list.
For each domain, we fetched https://[domain]/robots.txt. Of the 1,000 sites, 586 returned a valid robots.txt file. The other 414 returned errors, timeouts, or did not have a robots.txt at all. This is normal. Many domains in the top 1,000 are international sites, parked domains, or app domains that do not serve a traditional website.
We parsed each robots.txt file and checked it against a registry of 22 known AI crawler user-agents. This registry is the same one that powers the Parceit audit tool. It includes critical bots that directly power AI search products (GPTBot for ChatGPT, PerplexityBot for Perplexity, ClaudeBot for Claude, Google-Extended for Gemini and AI Overviews), major bots that feed AI training pipelines (CCBot for Common Crawl, Applebot-Extended for Apple Intelligence, Meta-ExternalAgent for Meta AI), and emerging bots from newer AI search products (YouBot, Cohere-ai, Diffbot).
A bot counts as 'blocked' when its robots.txt section contains Disallow: /. A bot not mentioned in a site's robots.txt defaults to 'allowed,' which is standard robots.txt behavior.
The headline numbers
Of the 586 sites with a valid robots.txt:
- 150 sites (25.6%) block at least one critical AI search crawler
- 21 sites (3.6%) block every critical AI search crawler we track
- 282 sites (48.1%) mention at least one AI crawler by name in their robots.txt, whether to block or allow it
That last number matters. Nearly half of major websites have made a deliberate decision about at least one AI crawler. This is not accidental blocking anymore. AI crawler policy is now a mainstream concern.
Which AI crawlers get blocked most
**CCBot (Common Crawl) - blocked by 23.7% of sites (139 sites).** CCBot is the most-blocked AI crawler in our dataset. It powers Common Crawl, the open dataset that trains many open-source AI models. Sites that block CCBot are opting out of contributing to open AI training data.
**ClaudeBot (Anthropic) - blocked by 20.6% of sites (121 sites).** Claude is blocked more than any other direct AI search crawler. One in five major sites does not want Claude crawling their content.
**GPTBot (OpenAI) - blocked by 20.8% of sites (122 sites).** GPTBot powers ChatGPT's web access and search. More than one in five sites blocks it. If ChatGPT cannot crawl your site, it cannot cite you in its answers.
**Google-Extended (Google) - blocked by 19.3% of sites (113 sites).** This one is interesting. Google-Extended controls whether Google can use your content for Gemini and AI Overviews. Nearly 20% of sites have opted out of Google's AI features specifically.
**PerplexityBot - blocked by 17.2% of sites (101 sites).** Perplexity's crawler is blocked by over 100 of the top 586 sites. Perplexity also runs Perplexity-User for live page fetches, which is blocked by 14.7% of sites.
**OAI-SearchBot (OpenAI) - blocked by 11.8% of sites (69 sites).** This is OpenAI's newer crawler specifically for ChatGPT Search live results. It is blocked less than GPTBot, probably because many sites have not updated their robots.txt to include it yet.
**Bingbot (Microsoft) - blocked by 4.3% of sites (25 sites).** Bingbot is rarely blocked. Only 25 sites in our dataset block it. Sites cannot risk losing visibility in Bing and Microsoft Copilot. Bingbot also crawls for traditional search results, not just AI features, so blocking it has broader consequences.
The training data war
When you look at the bots that feed AI training (not live search), the blocking is even more aggressive:
- CCBot (Common Crawl): 23.7% blocked
- Bytespider (ByteDance/TikTok): 22.4% blocked
- Meta-ExternalAgent (Meta AI): 19.5% blocked
- Applebot-Extended (Apple Intelligence): 17.7% blocked
- anthropic-ai (Claude training): 16.7% blocked
Sites are more willing to block training crawlers than live search crawlers. This suggests a split strategy: some sites want to appear in AI search results but do not want their content used to train competing models. That is a rational position, and it is becoming common.
Who is blocking everything
Three sites in our dataset block all 22 AI crawlers we track:
- reddit.com - blocks every AI bot including Bingbot. Reddit allows Googlebot but blocks Google-Extended, meaning Google can index Reddit for traditional search but cannot use the content for AI training.
- drive.google.com - blocks everything. This makes sense for a private file storage service.
- baidu.com - the Chinese search engine blocks all Western AI crawlers.
Other notable heavy blockers include x.com (21 of 22 bots blocked), yelp.com (21), facebook.com (20), quora.com (19), nytimes.com (16), cnn.com (17), wsj.com (17), reuters.com (18), and bbc.com (16).
The pattern is clear. Social media platforms and news organizations are the most aggressive blockers. They have content worth protecting, and they are willing to sacrifice AI search visibility to protect it.
What this means for your site
If you run a business that depends on being found in search, blocking AI crawlers is probably the wrong move. Here is why.
AI Overviews now appear in 43% of Google searches (per Similarweb data published July 2026). When a user searches for your product or service, Google increasingly generates an AI answer instead of showing ten blue links. If Google-Extended is blocked, your site cannot appear in that answer.
ChatGPT has over 200 million weekly users. If GPTBot is blocked, ChatGPT cannot recommend your business when users ask for suggestions in your category.
Perplexity is growing fast as a research tool for professionals, students, and journalists. If PerplexityBot is blocked, you are invisible there too.
The question is not whether AI search matters. It does, and it is growing. The question is whether your site is accessible to the crawlers that power it.
How to check your own site
You can check your robots.txt in 30 seconds.
- Open your browser and go to yourdomain.com/robots.txt
- Read the file. Look for any User-agent: lines that mention AI bots
- Look for Disallow: / under those lines. That means the bot is blocked
If you see bots you do not recognize listed with Disallow: /, look them up. GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and CCBot are the ones that matter most for AI visibility.
You can also run a free audit at parceit.com that checks your site against all 22 AI crawlers and tells you exactly which ones are blocked, which are allowed, and what the impact is.
Methodology
- Data collection date: July 31, 2026
- Source list: Tranco top-1 million domain list (daily snapshot)
- Sample: Top 1,000 domains from the Tranco list
- Fetch method: We requested https://[domain]/robots.txt for each site using a custom research user-agent. Sites that returned HTTP 200 with a valid text response were included. Sites that returned errors, timeouts, or no response were excluded.
- Bot registry: 22 AI crawler user-agents organized in three tiers (critical, major, emerging).
- Parsing logic: For each robots.txt file, we identified User-agent sections and checked whether each AI bot had a section containing Disallow: /. A bot with no specific section inherits rules from any User-agent: * wildcard section.
- Limitations: robots.txt is a policy declaration, not a technical enforcement mechanism. Some sites may block AI crawlers through WAF rules, Cloudflare challenges, or server-level filtering that would not appear in robots.txt. Our numbers represent declared policy only.
We plan to re-run this study periodically to track how AI crawler blocking changes over time.