Someone asked a question that most site owners never think to ask: is there a way to see which parts of my website AI crawlers like GPTBot and ClaudeBot are actually reading versus ignoring?
Yes. Your server access logs have the answer. Every time GPTBot, ClaudeBot, or PerplexityBot requests a page from your site, that request gets logged. The log entry shows which crawler visited, which URL it fetched, when it happened, and what response your server sent back.
Most people never look at these logs. They check robots.txt, confirm AI crawlers are allowed, and assume everything is fine. But "allowed" and "actually crawling" are different things. A crawler can be allowed in your robots.txt and still ignore most of your pages. Or it can be hitting pages that return errors. You will not know unless you look at the logs.
Here is how to do it.
Where crawler activity lives
Your web server records every HTTP request it receives. Whether you run Apache, Nginx, a CDN like Cloudflare, or a managed host like WP Engine, every visit gets an entry in an access log. That entry includes the visitor's user-agent string, which identifies what made the request.
AI crawlers identify themselves with specific user-agent strings. GPTBot uses "GPTBot." ClaudeBot uses "ClaudeBot." PerplexityBot uses "PerplexityBot." Google-Extended requests come from Googlebot with the Google-Extended flag, which shows up differently depending on your log format.
When you filter your access logs for these user-agents, you see exactly which pages each crawler requested, when it requested them, and what your server sent back.
Step 1: Get your access logs
Where your logs live depends on your hosting setup.
If you manage your own server (VPS, dedicated, or cloud instance), your access logs are on the server itself. Apache typically stores them at /var/log/apache2/access.log or /var/log/httpd/access_log. Nginx stores them at /var/log/nginx/access.log.
If you use a managed WordPress host like WP Engine, Kinsta, or SiteGround, you can download access logs from your hosting dashboard. Look for a section called "Logs," "Analytics," or "Access Logs." Most managed hosts keep 30 to 90 days of log history.
If you use Cloudflare, you can filter your Cloudflare logs or analytics for specific user-agents. Cloudflare's dashboard also shows bot traffic summaries, though the free plan limits how far back you can look.
If you use Vercel, Netlify, or another static host, you may not have traditional access logs. In that case, check whether your provider offers request analytics or logging as an add-on. Vercel includes request logs on Pro and Enterprise plans.
If none of these apply and you cannot access your server logs, skip to the section below on using a log monitoring tool.
Step 2: Filter for AI crawler user-agents
Once you have your access log file, filter it for AI crawler activity. The fastest way is from the command line.
This command shows every GPTBot request in your log:
grep "GPTBot" access.logDo the same for the other AI crawlers:
grep "ClaudeBot" access.log
grep "PerplexityBot" access.log
grep "Perplexity-User" access.logIf you want all AI crawler activity in one pass:
grep -E "GPTBot|ClaudeBot|PerplexityBot|Perplexity-User" access.logEach line in the output represents one HTTP request. A typical log entry looks something like this:
66.249.70.5 - - [04/Aug/2026:03:12:44 +0000] "GET /blog/how-to-get-cited-by-perplexity HTTP/2.0" 200 14532 "-" "GPTBot/1.2 (+https://openai.com/gptbot)"The important parts: the URL after GET, the 200 status code (meaning the page loaded successfully), and GPTBot at the end identifying itself.
If you see a 404, the crawler tried to reach a page that does not exist. A 403 means something blocked the request. A 301 or 302 means the page redirected. All of these matter.
Step 3: See which pages each crawler actually visited
Raw log output is hard to read. Extract just the URLs that each crawler requested so you can see the pattern.
This command lists every URL GPTBot fetched, sorted alphabetically:
grep "GPTBot" access.log | awk '{print $7}' | sort | uniq -c | sort -rnThe output shows the number of hits per URL. You will immediately see whether the crawler is crawling your whole site or just hitting a handful of pages.
A healthy crawl pattern looks like this: the homepage, your main content pages, your blog posts, and your internal links. The crawler is discovering and fetching content across your site.
An unhealthy pattern looks like this: the crawler hits your homepage and maybe one or two other pages, then stops. Or it only fetches pages that redirect. Or it hits the same page repeatedly without branching out. This means the crawler can reach your site but is not discovering your content.
The most common cause of a shallow crawl is poor internal linking. If your homepage does not link to your content pages, or if your navigation is JavaScript-rendered and the crawler cannot parse it, the crawler has no path to follow. AI crawlers, like all crawlers, discover pages by following links. If the links are not in the initial HTML response, the crawler does not see them.
Step 4: Identify what the crawlers are ignoring
Compare the list of URLs that GPTBot or ClaudeBot crawled against your full sitemap. The gap between the two is what the crawlers are ignoring.
Export your sitemap URLs. Most sites have one at yourdomain.com/sitemap.xml. If you use WordPress with Yoast or Rank Math, you have one already.
Then compare. If your sitemap lists 50 pages and GPTBot only crawled 8 of them, you know exactly which 42 pages are being ignored. The question is why.
There are a few common reasons.
The page is too deep in your site architecture. If a page is four or five clicks from the homepage, crawlers may never reach it. Move important content closer to the surface. Link to it from your homepage or main navigation.
The page is slow to load. Crawlers have crawl budgets. If your server takes five seconds to respond, the crawler moves on before fetching everything. Page speed matters for crawling, not just for user experience.
The page has no internal links pointing to it. An orphan page with no inbound links from your own site is effectively invisible to crawlers. Add at least one internal link from a page that crawlers already visit.
The page was added recently and the crawler has not revisited yet. AI crawlers do not crawl every day. GPTBot may check your site every few days or every few weeks depending on how often your content changes. New pages take time to be discovered.
Step 5: Check response codes for problems
Filter your log entries for error responses. This catches pages that crawlers are trying to reach but cannot.
grep "GPTBot" access.log | awk '{print $9}' | sort | uniq -c | sort -rnThis shows the distribution of HTTP status codes GPTBot received. If you see a lot of 404s, crawlers are trying to reach pages that do not exist. Check whether those URLs are in your sitemap or linked from your navigation. A sitemap full of URLs that return 404 wastes crawl budget and signals poor site maintenance.
If you see 403s, something is blocking the crawler on those specific paths even though your robots.txt allows it. This can happen when a security plugin, a firewall rule, or a CDN bot-protection feature is more aggressive than your robots.txt. Cloudflare's "Bot Fight Mode" is a frequent culprit. It can block AI crawlers at the edge even when your origin server would allow them.
Tools if you do not want to use the command line
If command-line log parsing is not your thing, several tools make this easier.
GoAccess is a free, open-source log analyzer that runs in your terminal and produces a real-time dashboard of your traffic, broken down by user-agent, URL, status code, and more. You can filter for specific crawlers. It installs in seconds on most Linux systems.
Screaming Frog Log File Analyser is a desktop tool (paid, with a free tier) that lets you upload your access log and filter by user-agent. It produces visual reports showing which pages each crawler hit, response codes, and crawl frequency. Useful if you prefer a GUI.
If you use Cloudflare, its analytics dashboard shows bot traffic by category. You can see how many requests came from verified bots (which includes major AI crawlers) without touching a log file.
Google Search Console does not show AI crawler activity. It only shows Googlebot. For GPTBot, ClaudeBot, and PerplexityBot, you need your own server logs or a log analysis tool.
What to do with what you find
Once you know which pages AI crawlers are reading and which they are ignoring, the fixes are usually straightforward.
Fix internal linking. Make sure every important page is reachable within two or three clicks from the homepage. If you published a blog post three months ago and no crawler has ever fetched it, the problem is almost certainly that no page links to it.
Fix your sitemap. If your sitemap includes URLs that return 404, remove them. If it is missing pages you want crawled, add them. Submit the updated sitemap in Google Search Console and at the webmaster tools for each AI engine that offers one.
Check for crawl blocks beyond robots.txt. A robots.txt Allow rule does not help if Cloudflare, your firewall, or a security plugin is blocking the crawler before it reaches your origin server. Test access directly with curl:
curl -A "GPTBot" -I https://yourdomain.com/blog/your-articleA 200 response means the crawler can reach the page. Anything else means there is a block somewhere in your stack.
Be patient. After you fix linking or sitemap issues, the crawler needs to revisit your site to discover the changes. This takes days to weeks depending on the crawler's schedule.
Why this matters in the IAB framework
The IAB published its "Measuring Visibility in the AI Era" framework on August 3, 2026. It defines the first P of AI Visibility as Presence: does your brand appear in AI responses at all? Presence depends on crawlability. If AI crawlers cannot read your pages, your Presence metrics are zero.
But crawlability has two layers. The first layer is whether your robots.txt allows the crawler. The second layer, the one most people skip, is whether the crawler actually reaches and fetches your content. Server log analysis is how you verify the second layer. Without it, you are assuming the crawler is doing what your robots.txt says it may do.
In IAB terms, checking your logs is the difference between directional and decision-grade measurement. Saying "my robots.txt allows GPTBot" is directional. Saying "GPTBot crawled 42 of my 50 pages last month and every page returned a 200" is decision-grade. It is specific, reproducible, and actionable.
Frequently asked questions
Can I see which pages GPTBot is reading on my website?
Yes. Your server access logs record every request GPTBot makes. Filter your logs for the "GPTBot" user-agent string and you will see every URL it fetched, when it fetched it, and what response code your server returned. This works for ClaudeBot, PerplexityBot, and any other crawler with an identifiable user-agent.
Does Google Search Console show AI crawler activity?
No. Google Search Console only reports on Googlebot activity. It does not show requests from GPTBot, ClaudeBot, PerplexityBot, or other AI crawlers. To see those, you need your own server access logs or a log analysis tool.
Why would an AI crawler be allowed in robots.txt but not crawling my pages?
Several reasons. Your internal linking may be too shallow, so the crawler cannot discover pages. Your pages may be slow to load, causing the crawler to hit its crawl budget. Your sitemap may contain broken links. A CDN or security plugin may be blocking the crawler before it reaches your server. Server log analysis tells you which of these is the actual problem.
How often do AI crawlers visit a website?
It varies by crawler and by site. Sites that update frequently and have high crawl rates get visited more often. GPTBot and PerplexityBot typically revisit every few days to every few weeks for sites they already know about. Sites that were previously blocked or are newly discovered may wait longer for a first crawl. Check your access logs to see the actual cadence for your site.