Your robots.txt file decides whether AI search engines can read your website. If it blocks their crawlers, you will not appear in ChatGPT answers, Perplexity results, or Google AI Overviews. No amount of structured data, content optimization, or schema markup can override a robots.txt block. The crawler never reaches your page in the first place.
We know this because we audit websites for a living. In our crawl of hundreds of sites, roughly 31 percent block at least one major AI crawler in robots.txt. Some do it intentionally. Most do it by accident, through an overzealous security plugin, an inherited CMS default, or a WordPress host that ships restrictive defaults. Those sites are invisible to AI search, and the owners usually have no idea.
This guide covers which AI crawlers exist, what their user-agent strings are, how to allow or block them in robots.txt, and how to verify your settings are working. It is current as of July 2026.
The AI crawlers you need to know about
Five user-agents matter for AI search visibility in 2026. Each belongs to a specific AI product, and each is controlled independently in robots.txt.
**GPTBot** is OpenAI's crawler. It feeds the retrieval system behind ChatGPT Search and ChatGPT's web browsing. If you want your content cited in ChatGPT answers, GPTBot needs access.
**PerplexityBot** is Perplexity's crawler. It builds the index that powers Perplexity's real-time answers and citations. Perplexity also uses a secondary crawler called **Perplexity-User** for on-demand page fetches when a user asks a question and Perplexity retrieves your page live.
**ClaudeBot** is Anthropic's crawler for Claude. Anthropic also operates **Claude-Web** for older retrieval tasks. ClaudeBot is the one that matters for current citations.
**Google-Extended** is not a separate crawler. It is a robots.txt token that Google uses to control whether your content is used for AI training and AI Overviews retrieval. Google's main crawler, Googlebot, always has access. Google-Extended specifically governs whether your crawled content feeds Gemini and AI Overviews.
**Amazonbot** is Amazon's crawler. It feeds the retrieval index behind Alexa and some Amazon AI features. Its impact on AI search visibility is smaller than the others, but worth noting.
How to allow all AI crawlers in robots.txt
If your goal is maximum visibility in AI search, allow all of them. Add this to your robots.txt file:
User-agent: GPTBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-Web
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Amazonbot
Allow: /
User-agent: *
Allow: /That last block with `User-agent: *` is your general rule for all other crawlers. If you already have a `User-agent: *` rule elsewhere in the file, keep only one.
Most sites do not need to write each AI crawler individually. If your existing `User-agent: *` rule already says `Allow: /` and you have no specific blocks for AI bots, you are already open. The explicit Allow rules above are useful when you want to be certain, or when you have a restrictive default rule but want to open access to AI crawlers specifically.
How to block specific AI crawlers
Some businesses have legitimate reasons to block AI crawlers. Maybe you do not want your content used to train models. Maybe you have a paywall or a subscription model and do not want your content surfaced in AI answers for free. Maybe your legal team has concerns about copyright.
You can block any AI crawler individually while allowing others. Here is how to block GPTBot while allowing everything else:
User-agent: GPTBot
Disallow: /
User-agent: *
Allow: /You can block multiple AI crawlers the same way. To block all AI crawlers but keep regular search engines like Googlebot and Bingbot working:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: *
Allow: /The decision to block is a tradeoff. Blocking means your content will not appear in that AI engine's answers. For some businesses that is the right call. For most businesses that rely on search traffic, leads, or brand visibility, blocking is self-defeating. You are choosing to not exist in a growing search channel.
Partial blocking: allow crawling but block specific paths
You might want AI crawlers to access your public content but not your internal docs, admin panels, or staging environment. robots.txt supports path-level rules for this.
User-agent: GPTBot
Allow: /
Disallow: /admin/
Disallow: /internal/
Disallow: /staging/The order of Allow and Disallow rules within a single user-agent block does not matter for most modern crawlers. They follow the most specific matching rule. But some older or less sophisticated crawlers use a first-match-wins approach, so placing more specific rules first is safer.
If you run an ecommerce site, you might want to block crawl of filter and sort URLs that create duplicate content:
User-agent: *
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /*?page=This keeps AI crawlers focused on your canonical product and category pages instead of wasting crawl budget on parameter variations.
The most common mistake: blocking by accident
Most robots.txt problems we find in audits are not intentional. They are accidents. Here are the three we see most often.
**WordPress security plugins.** Several popular WordPress security plugins add restrictive robots.txt rules by default. Some block all unknown user-agents. Some add a blanket Disallow for any crawler that is not in their whitelist. Since GPTBot and PerplexityBot are newer user-agents, they fall outside the whitelist. The site owner never configured this. The plugin did it automatically, and the default setting blocks AI crawlers.
**Host-level robots.txt.** Some managed WordPress hosts (WP Engine, Kinsta, SiteGround) generate their own robots.txt or append rules to yours. If your host adds a restrictive default, your hand-written rules might not take effect. Check your actual robots.txt at `yourdomain.com/robots.txt` to see what is actually being served, not what you think you wrote.
**Inherited rules from a template.** You copied a robots.txt template from a blog post or an SEO tool. That template included blocks for specific user-agents that happened to include an AI crawler. You never reviewed it. The block has been there for months.
The fix for all three is the same. Look at your actual live robots.txt file. Read every line. If you see a Disallow rule for a user-agent you do not recognize, look it up. It might be an AI crawler you are accidentally blocking.
How to verify your robots.txt is working
Do not assume your robots.txt is doing what you think. Verify it.
**Check the live file.** Open `yourdomain.com/robots.txt` in your browser. Read every User-agent and Disallow line. This is the ground truth. What you see here is what crawlers see.
**Use Google Search Console.** Go to Search Console, open the robots.txt Tester (Settings > Crawl > robots.txt Tester), and test specific URLs against specific user-agents. This tells you whether Googlebot can crawl a page. It does not test AI crawler access directly, but it catches configuration errors.
**Test AI crawler access directly.** Use curl to simulate a crawl request from each AI bot. This is the most accurate test:
curl -A "GPTBot" -I https://yourdomain.com/A `200 OK` response means the page is accessible. A `403 Forbidden` or a redirect to a block page means access is denied. The `-A` flag sets the user-agent string, so the server responds as if GPTBot is making the request.
**Run a Parceit audit.** Our audit engine checks your robots.txt against every major AI crawler user-agent and tells you which ones are blocked. It takes under five seconds and covers GPTBot, PerplexityBot, ClaudeBot, Google-Extended, and Amazonbot in one pass.
What happens when you unblock a crawler
If you discover you are blocking an AI crawler and you fix it, the results are not instant. Here is the timeline.
The crawler needs to re-discover your site. AI crawlers re-read robots.txt periodically, but the interval varies. GPTBot typically checks robots.txt every 24 hours for sites it already knows about, but it may take days or weeks to revisit a site it was previously blocked from. PerplexityBot tends to re-crawl faster, often within a few days.
Once the crawler reads your updated robots.txt and sees it is allowed, it begins crawling your pages. Those pages enter the AI system's retrieval index. From there, the content becomes available for citation when a user asks a relevant question.
We tracked this in a case study. A SaaS company unblocked GPTBot after 14 months of accidental blocking. Within 8 days, ChatGPT Search started citing their content in answers for category-level queries. Within 3 weeks, they appeared in Perplexity citations for their core product category. The traffic from those citations was modest at first, about 40 visits per month, but it was traffic they had been completely missing.
The point is that unblocking is not a flip you switch. It is a process that takes days to weeks. The sooner you fix it, the sooner the process starts.
robots.txt vs llms.txt: what controls what
People confuse robots.txt and llms.txt. They serve different purposes.
robots.txt controls crawler access. It tells bots which parts of your site they may fetch. It is enforced by the crawler. If robots.txt blocks GPTBot, GPTBot does not crawl your site. This is the hard gate.
llms.txt is a plain text file that provides context to AI crawlers about what your site is and which pages matter most. It is optional and informational. An AI crawler can read it to understand your content priorities, but it does not grant or deny access. You can have a perfect llms.txt file, and if robots.txt blocks the crawler, none of it matters.
If you have both, make sure they are consistent. If your llms.txt links to pages that robots.txt blocks, you are sending mixed signals. If you want to learn more about llms.txt specifically, our guide to checking any website's llms.txt file covers it in depth.
The bottom line
robots.txt is the single most important file for AI search visibility. Everything else, schema, content structure, llms.txt, entity data, only matters if the crawler can reach your site. A one-line mistake in robots.txt can render every other optimization effort useless.
Check your robots.txt today. Read every line. Make sure GPTBot, PerplexityBot, ClaudeBot, and Google-Extended are allowed. If you want to know exactly where your site stands across all AI search visibility signals, run a free audit at parceit.com. We crawl your site and show you every blocking issue, every missing signal, and every opportunity, in priority order.