A claim made the rounds this week that Amazon had unblocked its AI crawlers. We checked. It is false, and the real story in AI shopping is bigger than a robots.txt edit.
Parceit fetches and parses robots.txt files from the 1,000 biggest websites every month. On September 1, amazon.com blocked 16 of the 22 AI crawlers we track, exactly as it did on August 1. We also pulled the live file today: GPTBot, ClaudeBot, CCBot, Google-Extended and Bytespider all get "Disallow: /", a full-site block. There is no Allow rule for any AI crawler in the file.
Where the rumor came from
One real un-block did happen in our data, and it was Facebook. It removed its blanket "Disallow: /" for six bots including GPTBot, ClaudeBot, PerplexityBot, Google-Extended and Amazonbot. Those bots now see partial path restrictions instead of a full block. "Facebook unblocked Amazonbot" is exactly the kind of sentence that turns into "Amazon unfollowed" by the time it reaches a group chat.
The direction of travel at Amazon is the opposite. Modern Retail reported this summer that Amazon added six more AI crawlers to its blocklist, covering bots from Meta, Google, Huawei and Mistral, after earlier blocks on Claude, Perplexity and Google's Project Mariner agents. The change was first spotted by Juozas Kaziukėnas, an independent analyst. "Amazon is desperately trying to stop AI companies from training models on its data," he wrote. "But Amazon is definitely not interested in helping anyone build the future of AI shopping. If that is indeed the future, Amazon wants to build it itself."
The actual news: checkout is moving into AI search
On January 11, Google launched the Universal Commerce Protocol (UCP), an open standard for agentic commerce co-developed with Shopify, Etsy, Wayfair, Target and Walmart, and endorsed by more than 20 companies including Visa, Mastercard, American Express, Stripe, Adyen, Best Buy, Macy's, The Home Depot and Zalando. It is built to work alongside existing agent protocols including A2A, AP2 and MCP.
In Google's words: "UCP will soon power a new checkout feature on eligible Google product listings in AI Mode in Search and the Gemini app, allowing shoppers to check out from eligible U.S. retailers right as they're researching on Google."
Google also switched on Business Agent, a branded chat experience inside Search, with retailers including Lowe's, Michael's, Poshmark and Reebok at launch. Direct purchases, including agentic checkout, are promised in the coming months, alongside dozens of new Merchant Center data attributes built for conversational discovery and a new Direct Offers ad format.
OpenAI publishes its own Agentic Commerce Protocol, defining calls like search_products, add_to_cart and checkout that let a shopping agent transact with a merchant directly. Perplexity already runs an AI shopping assistant. Shopify now sets policy in its merchants' robots.txt files for "buy-for-me" agents, requiring a human review step, and has partnered with Perplexity on transactions.
Amazon's incentive, in dollars
Why the wall? Modern Retail points to the money: Amazon's marketplace is the foundation of a $56 billion-a-year advertising business built on shoppers browsing its site. Outside AI tools that surface products directly bypass the storefront, undermining both traffic and ad revenue. So Amazon blocks the bots, keeps building its own shopping chatbot, Rufus, with ads, and tests its own buy-for-me feature for purchasing from other retailers' sites.
robots.txt is advisory. Well-behaved crawlers respect it. Nothing stops a determined scraper, and much of Amazon's catalog is already inside existing training datasets. The block is a signpost of intent: AI shopping is welcome when Amazon builds it.
What 1,000 sites say about where merchants actually stand
- 59.0 percent of the top 1,000 sites have a robots.txt file at all (590 of 1,000)
- 26.3 percent of those block at least one critical AI search crawler (155 of 590)
- 4.1 percent block every critical AI search crawler (24 of 590)
- The most-blocked bots are CCBot at 24.4 percent, Bytespider at 23.4 percent, ClaudeBot at 21.7 percent, GPTBot at 21.4 percent and Google-Extended at 19.7 percent
Month over month, blocking ticked up slightly across almost every bot we track. Zero sites in our sample went from blocking at least one critical AI search bot to blocking none. The one major opener was Facebook, and nobody followed it. Most of the web is still open to AI search crawlers. The walls are concentrated at the biggest properties.
The claim that Amazon had unblocked its AI crawlers is one more reminder of why primary verification matters. The live file is the ground truth. The live file still says no.
What to do this week
- Pull up your robots.txt and separate search-index bots (OAI-SearchBot, PerplexityBot, GPTBot, ClaudeBot) from training bots (CCBot, Bytespider, anthropic-ai). Visibility in AI answers runs through the first group.
- If you sell on Google, claim your Merchant Center listings now. The new conversational attributes and the UCP checkout rollout both ride on that feed.
- Finish your Product schema: price, availability, ratings, compatible accessories. Agents read structured data before they read prose, and the same fields map onto what OpenAI's protocol asks for.
- Decide your stance on buy-for-me agents before one shows up in your logs. Shopify merchants already have a default policy whether they know it or not.
The AI shopping market is splitting into an open rail and a walled garden. Google and OpenAI are laying track for agents to discover and buy. Amazon is closing the gate and building its own train. Either way, the ticket is the same: clean, complete, machine-readable product data on a site the search bots can reach.
Sources: Parceit top-1,000 robots.txt study, September 1, 2026 (live fetches); Google, "New tech and tools for retailers to succeed in an agentic shopping era," January 11, 2026; Modern Retail, "Amazon quietly blocks AI bots from Meta, Google, Huawei and more"; OpenAI Agentic Commerce Protocol documentation.