Your robots.txt file decides whether AI search engines can read your website. If it blocks their crawlers, you will not appear in ChatGPT answers, Perplexity results, or Google AI Overviews. No amount of structured data, content optimization, or schema markup can override a robots.txt block. The crawler never reaches your page in the first place.
Many sites unknowingly block at least one major AI crawler in robots.txt. Some do it intentionally. Most do it by accident, through an overzealous security plugin, an inherited CMS default, or a WordPress host that ships restrictive defaults. Those sites are invisible to AI search, and the owners usually have no idea.
This guide covers which AI crawlers exist, what their user-agent strings are, how to allow or block them in robots.txt, and how to verify your settings are working. It is current as of August 2026, and the newest crawlers are covered in the update at the end of this guide.
The AI crawlers you need to know about
Seven AI crawler names matter for AI search visibility in 2026, and the list keeps growing. Every time a crawler requests a page from your site, it identifies itself by name. That name is the user-agent, and robots.txt rules work by matching those names. The seven names belong to five companies: OpenAI, Anthropic, Google, Perplexity, and Amazon. Newer agents, including OAI-SearchBot, Claude-SearchBot, and Meta's crawlers, are covered in the August 2026 update at the end of this guide.
GPTBot is OpenAI's crawler. It feeds the retrieval system behind ChatGPT Search and ChatGPT's web browsing. If you want your content cited in ChatGPT answers, GPTBot needs access.
PerplexityBot is Perplexity's crawler. It builds the index that powers Perplexity's real-time answers and citations. Perplexity also uses a secondary crawler called Perplexity-User for on-demand page fetches when a user asks a question and Perplexity retrieves your page live.
ClaudeBot is Anthropic's crawler for Claude. Anthropic also operates Claude-Web for older retrieval tasks. ClaudeBot is the one that matters for current citations.
Google-Extended is not a separate crawler. It is a robots.txt token that Google uses to control whether your content is used for AI training and AI Overviews retrieval. Google's main crawler, Googlebot, always has access. Google-Extended specifically governs whether your crawled content feeds Gemini and AI Overviews.
Amazonbot is Amazon's crawler. It feeds the retrieval index behind Alexa and some Amazon AI features. Its impact on AI search visibility is smaller than the others, but worth noting.
How to allow all AI crawlers in robots.txt
If your goal is maximum visibility in AI search, allow all of them. Add this to your robots.txt file:
User-agent: GPTBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-Web
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Amazonbot
Allow: /
User-agent: *
Allow: /
That last block with User-agent: * is your general rule for all other crawlers. If you already have a User-agent: * rule elsewhere in the file, keep only one.
Most sites do not need to write each AI crawler individually. If your existing User-agent: * rule already says Allow: / and you have no specific blocks for AI bots, you are already open. The explicit Allow rules above are useful when you want to be certain, or when you have a restrictive default rule but want to open access to AI crawlers specifically.
How to block specific AI crawlers
Some businesses have legitimate reasons to block AI crawlers. Maybe you do not want your content used to train models. Maybe you have a paywall or a subscription model and do not want your content surfaced in AI answers for free. Maybe your legal team has concerns about copyright.
You can block any AI crawler individually while allowing others. Here is how to block GPTBot while allowing everything else:
User-agent: GPTBot
Disallow: /
User-agent: *
Allow: /
You can block multiple AI crawlers the same way. To block all AI crawlers but keep regular search engines like Googlebot and Bingbot working:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: *
Allow: /
The decision to block is a tradeoff. Blocking means your content will not appear in that AI engine's answers. For some businesses that is the right call. For most businesses that rely on search traffic, leads, or brand visibility, blocking is self-defeating. You are choosing to not exist in a growing search channel.
Partial blocking: allow crawling but block specific paths
You might want AI crawlers to access your public content but not your internal docs, admin panels, or staging environment. robots.txt supports path-level rules for this.
User-agent: GPTBot
Allow: /
Disallow: /admin/
Disallow: /internal/
Disallow: /staging/
The order of Allow and Disallow rules within a single user-agent block does not matter for most modern crawlers. They follow the most specific matching rule. But some older or less sophisticated crawlers use a first-match-wins approach, so placing more specific rules first is safer.
If you run an ecommerce site, you might want to block crawl of filter and sort URLs that create duplicate content:
User-agent: *
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /*?page=
This keeps AI crawlers focused on your canonical product and category pages instead of wasting crawl budget on parameter variations.
The most common mistake: blocking by accident
Most robots.txt problems we find in audits are not intentional. They are accidents. Here are the three we see most often.
WordPress security plugins. Several popular WordPress security plugins add restrictive robots.txt rules by default. Some block all unknown user-agents. Some add a blanket Disallow for any crawler that is not in their whitelist. Since GPTBot and PerplexityBot are newer user-agents, they fall outside the whitelist. The site owner never configured this. The plugin did it automatically, and the default setting blocks AI crawlers.
Host-level robots.txt. Some managed WordPress hosts (WP Engine, Kinsta, SiteGround) generate their own robots.txt or append rules to yours. If your host adds a restrictive default, your hand-written rules might not take effect. Check your actual robots.txt at yourdomain.com/robots.txt to see what is actually being served, not what you think you wrote.
Inherited rules from a template. You copied a robots.txt template from a blog post or an SEO tool. That template included blocks for specific user-agents that happened to include an AI crawler. You never reviewed it. The block has been there for months.
[WARNING] The fix for all three is the same. Look at your actual live robots.txt file. Read every line. If you see a Disallow rule for a user-agent you do not recognize, look it up. It might be an AI crawler you are accidentally blocking.
How to verify your robots.txt is working
Do not assume your robots.txt is doing what you think. Verify it.
Check the live file. Open yourdomain.com/robots.txt in your browser. Read every User-agent and Disallow line. This is the ground truth. What you see here is what crawlers see.
Use Google Search Console. Go to Search Console, open the robots.txt Tester (Settings > Crawl > robots.txt Tester), and test specific URLs against specific user-agents. This tells you whether Googlebot can crawl a page. It does not test AI crawler access directly, but it catches configuration errors.
Test AI crawler access directly. Use curl to simulate a crawl request from each AI bot. This is the most accurate test:
curl -A "GPTBot" -I https://yourdomain.com/
A 200 OK response means the page is accessible. A 403 Forbidden or a redirect to a block page means access is denied. The -A flag sets the user-agent string, so the server responds as if GPTBot is making the request.
Run a Parceit audit. Our audit engine checks your robots.txt against every major AI crawler user-agent and tells you which ones are blocked. It takes under five seconds and covers GPTBot, PerplexityBot, ClaudeBot, Google-Extended, and Amazonbot in one pass.
What happens when you unblock a crawler
If you discover you are blocking an AI crawler and you fix it, the results are not instant. Here is the timeline.
The crawler needs to re-discover your site. AI crawlers re-read robots.txt periodically, but the interval varies. GPTBot typically checks robots.txt every 24 hours for sites it already knows about, but it may take days or weeks to revisit a site it was previously blocked from. PerplexityBot tends to re-crawl faster, often within a few days.
Once the crawler reads your updated robots.txt and sees it is allowed, it begins crawling your pages. Those pages enter the AI system's retrieval index. From there, the content becomes available for citation when a user asks a relevant question.
How long does this take in practice? Based on what AI search engines have publicly disclosed about their crawl schedules, expect days to weeks, not hours. GPTBot and PerplexityBot do not crawl on demand. They revisit sites on their own cadence. If your site was previously blocked, it may take longer for the crawler to rediscover you than if you were simply new. The sooner you fix the block, the sooner the clock starts.
The point is that unblocking is not a flip you switch. It is a process that takes days to weeks. The sooner you fix it, the sooner the process starts.
robots.txt vs llms.txt: what controls what
People confuse robots.txt and llms.txt. They serve different purposes.
robots.txt controls crawler access. It tells bots which parts of your site they may fetch. It is enforced by the crawler. If robots.txt blocks GPTBot, GPTBot does not crawl your site. This is the hard gate.
llms.txt is a plain text file that provides context to AI crawlers about what your site is and which pages matter most. It is optional and informational. An AI crawler can read it to understand your content priorities, but it does not grant or deny access. You can have a perfect llms.txt file, and if robots.txt blocks the crawler, none of it matters.
If you have both, make sure they are consistent. If your llms.txt links to pages that robots.txt blocks, you are sending mixed signals. If you want to learn more about llms.txt specifically, our guide to checking any website's llms.txt file covers it in depth.
The bottom line
robots.txt is the single most important file for AI search visibility. Everything else, schema, content structure, llms.txt, entity data, only matters if the crawler can reach your site. A one-line mistake in robots.txt can render every other optimization effort useless.
Check your robots.txt today. Read every line. Make sure GPTBot, PerplexityBot, ClaudeBot, and Google-Extended are allowed. If you want to know exactly where your site stands across all AI search visibility signals, run a free audit at parceit.com. We crawl your site and show you every blocking issue, every missing signal, and every opportunity, in priority order.
Update August 2026: robots.txt is no longer enough
Everything in this guide is still correct. But the landscape has shifted. In August 2026, Patreon abandoned robots.txt entirely and switched to Cloudflare to enforce crawler rules. This is a signal that robots.txt on its own is increasingly viewed as insufficient. Robots.txt works on the honor system. Crawlers can ignore it. Cloudflare enforcement actually blocks them at the network level.
Cloudflare replaced its old single "block AI bots" toggle with a three-category system based on crawler behavior: Search (retrieval for answers), Agent (actions on behalf of users), and Training (model training). The problem is that these categories overlap. Search Engine Journal reported that blocking Training crawlers can accidentally block Googlebot because the same crawler may fall into multiple categories.
If your site is behind Cloudflare and you block AI Training crawlers, check whether Search crawlers are also affected. The safest approach is to block selectively, page by page, rather than applying a blanket block that might catch crawlers you want to keep.
The advice in this guide remains your foundation. Allow GPTBot, PerplexityBot, ClaudeBot, and Google-Extended in robots.txt. But if your site uses Cloudflare or another WAF, also check your firewall-level bot rules. Robots.txt tells crawlers what to do. Cloudflare decides whether they can reach you at all.
Update August 2026: the crawler list keeps growing
The seven agents above are the core set. Three more now matter enough to know by name.
OAI-SearchBot is OpenAI's crawler for search. It is the agent that decides whether your content appears in ChatGPT Search results. ChatGPT-User is the separate agent that fetches pages when a user clicks. OpenAI has said user-initiated fetches may not be bound by robots.txt, and TollBit log data from August 2026 shows ChatGPT-User reaching disallowed URLs on a meaningful share of sites. If you allow GPTBot, allow OAI-SearchBot too.
Claude-SearchBot is Anthropic's newer search fetcher. As of mid-August 2026 it is explicitly listed in the robots.txt files of The Guardian, The New York Times, LinkedIn, Quora, Amazon, and CNBC. Anthropic says all of its bots respect robots.txt.
Meta-ExternalAgent and FacebookExternalAgent are Meta's crawlers. Meta is crawling the web at scale, and industry reporting ties the activity to building its own AI search. Meta crawlers were measured at over half of AI bot traffic on some networks. If Meta ships that search engine, sites that blocked these crawlers will be missing from its answers, so decide whether you want in.
Longer tail agents keep appearing, among them Gemini-Deep-Research, AI2Bot-DeepResearchEval, and KlaviyoAIBot. You do not need to track every one. Set two recurring checks and one tripwire. Re-read your own robots.txt once a quarter; the file only changes when you change it. Scan your server logs for user-agents you do not recognize every two weeks; new crawlers appear in your logs before they appear in any guide. The tripwire is the dated policy change: when a platform announces one, act before the date arrives. Cloudflare set the current example. Its September 15, 2026 update will treat Googlebot and Bingbot as AI training crawlers, which means a site using Cloudflare to block AI bots will block Google and Bing too unless it changes its settings first.
How fast you run these checks should match what your content is worth in licensing. If original content is the product, robots.txt is a negotiating position, and the large publishers revise theirs monthly: Reddit disallows all crawling by default and licenses access separately, while The Guardian, The New York Times, and LinkedIn each maintain explicit per-agent rules. If your site is a storefront or a service business, the quarterly review plus the two-week log scan keeps you current.