Four numbers frame this guide. Vercel and MERJ counted 569 million GPTBot fetches in one month and found that none ran JavaScript, so analytics never saw them (section 1). Anthropic crawled 1,782 pages for every referral it sent in mid 2026, against 4.79 for Google (section 4). About 35% of ChatGPT’s fetches hit a missing page, which a log shows and GA4 does not (section 7). And AI referred visits to US retailers converted 60% better in July 2026, which is why ThriveStack citedby helps you close the gaps (section 7).
Our research with Vercel highlights that AI crawlers, while rapidly scaling, continue to face significant challenges in handling JavaScript and efficiently crawling content.
Vercel, The Rise of the AI Crawler, Dec 17, 2024
Why does AI search visibility depend on server logs?
AI answers change on every run, so there is no rank to check, and analytics never sees the bots. Your server log is the one record of what an AI engine did with your site: which bot came, which page it asked for and which status code it got back. That makes AI search visibility something you can check.
The figure shows why the two tools disagree. GA4 and most marketing analytics count a visit only when a script runs in a browser. AI crawlers read the raw HTML and leave. Vercel and MERJ measured this on Vercel’s network in late 2024. In one month they saw 569 million fetches from OpenAI’s GPTBot and 370 million from Claude. They found that the bots from OpenAI, Anthropic, Perplexity, Meta and ByteDance do not render JavaScript. Google’s Gemini crawler and AppleBot did.
Because the tag never fires, the visit never reaches your dashboard. Tag managers, heatmaps and most attribution tools share the blind spot. A rank tracker has its own limit. It was built for ten blue links, and an AI answer has none.
The server log has none of these limits, because it records the request itself. The log is the receipt. It tells you whether an AI engine reached a page and what it got when it did. Peec AI makes the same case: logs show whether key bots are blocked or show little interest in a domain.
Read next: What GA4 can and cannot see about AI traffic.
How does ThriveStack citedby show what GPTBot did with your pages?
ThriveStack citedby reads your server logs, runs your tracked prompts through the AI engines and joins the two on each URL. You see which pages a bot fetched, which pages an answer cited and where the two do not match. Those mismatches are your citation gaps, ranked by what to fix first.
The figure shows the flow. Layer 1 is your own log, from a source such as Cloudflare, Vercel, WordPress or AWS. Layer 2 is synthetic prompt analysis: the questions your buyers ask, asked of ChatGPT, Perplexity and Gemini on a schedule, with the cited URLs recorded. The join happens on the page address, so each row tells one story.
A cited page shows a successful fetch and a citation. A gap shows a block, a skip or no fetch at all. That is the same reading this guide teaches by hand, done for every page at once. It is a shortcut for the method below, and you can follow the method without it.
What are the three types of AI crawlers in your logs?
There are three types: training bots, search bots and live fetch bots. Each does a different job and leaves a different mark in your log. Training bots feed the model. Search bots build the index that answers draw from. Live fetch bots grab one page for one prompt. Function matters more than the company name.
The figure sorts the bots you will see in a log. OpenAI, Anthropic and Perplexity each publish their bot lists, so you can group hits by name. OpenAI describes GPTBot as its training crawler and OAI-SearchBot as the one that surfaces sites in ChatGPT search. ChatGPT-User handles actions a person starts. Anthropic runs ClaudeBot for training, Claude-SearchBot for search quality and Claude-User for user questions. Perplexity splits PerplexityBot, which feeds its search results, from Perplexity-User, which handles user actions.
The split decides what a block costs you. Blocking a training bot keeps your pages out of model training. It leaves you in the answers. OpenAI says sites that opt out of OAI-SearchBot will not appear in ChatGPT search answers, and Anthropic says blocking Claude-SearchBot or Claude-User may reduce your visibility. So the wrong block quietly removes you from answers.
Check robots.txt, and check your CDN and firewall too. A bot rule at that layer can return a 403 that robots.txt never shows. Live fetch bots follow different rules. OpenAI says robots.txt rules “may not apply” to ChatGPT-User because a person asked for the page, and Perplexity says Perplexity-User generally ignores them. Anthropic says its bots honor robots.txt. A starter file that lets the citing bots in looks like this:
# Allow the bots that can cite you User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-User Allow: /
Read next: Is blocking AI bots always a bad idea?
What does log file analysis show that analytics cannot?
It shows every AI bot hit and the status code each one got, which GA4 and Search Console never record. A 200 means the bot read the page. A 403 or a timeout means it could not. No amount of good content fixes a block, so the status code is the first thing to read.
The log also shows the gap between crawling and visits. Cloudflare tracks pages crawled per referral, the crawl to refer ratio. In the 28 days to June 22, 2026, Anthropic crawled 4,580 pages for every referral it sent, OpenAI 848 and Google 5. By the 28 days to August 1 the figures were 1,782, 233 and 4.79. Perplexity went the other way, from 186 to 303. The ratios move between windows, so read them as a range. Cloudflare also warns they may overstate the gap, because native apps send no Referer header.
Training still dominates what bots fetch. In July 2025 Cloudflare put training at nearly 80% of AI bot crawling. Heavy crawling therefore says little about citations. It measures data collection. A citation needs an open path to your page and content that an engine chooses to use. Our fact-check on whether crawler logs guarantee citations covers this in more detail.
- The silent block. Perplexity-User asked for the page and the edge returned a 403. Fix access first, since no other work helps. See the setup section.
- Crawled, uncited. The 200 rules out access, and the bot still chose a rival. That points to content and structure. See the fixes section.
- Confirmed citation. A live fetch, then a person arriving from chatgpt.com moments later. Find more pages like it. See the method section.
What is citation gap analysis in AI search?
Citation gap analysis finds the prompts where competitors get cited and you do not. It splits two steps that look alike. Retrieval means the AI pulled your page as a candidate source. Citation means the AI named you in the answer. A page can be retrieved often and cited rarely.
The figure follows 1,000 sitemap pages down two paths. One goes through Google crawling and indexing, the other through AI bot hits and citations. The numbers are made up to show the shape. Search Console shows indexing, your logs show which AI bots live crawled a page, and a prompt tracker shows which pages got named. Each leak has its own fix: tune for indexing, for AI readability or for the prompts.
Peec AI describes the same gap. A page that bots request often and prompts never name is being considered but not selected. That is the richest place to win, because the content is already close to good enough. AI answers change from run to run, so read citation trends over weeks rather than one day. For the wider picture, see our research on AI citation sources in 2026 and the AI visibility gap report.
Google position does not tell you who gets cited. Ahrefs checked 15,000 long-tail prompts. Pages that Perplexity cited overlapped Google’s top 10 at 28.6%, and ChatGPT’s in-text citations at 8.0%. Across the engines Ahrefs reports, the average was about 11.9%. You cannot read your AI visibility off your Google rank.
See which of your pages AI can actually cite
Connect Google Search Console to ThriveStack citedby and the whole pipeline shows up in one view: every sitemap page, Google crawled vs indexed, the pages AI bots fetched in your server logs, and the citations they earned.
See the sample pipeline →Sample data, no signup. Connect Search Console when you are ready.How do you match AI crawler logs to AI citation gaps?
Cross two lists: what your logs say each bot did, and what your tracker says each prompt returned. The overlap sorts every page into one of five buckets, and each bucket has one fix. The logs and the tracker are each half the picture, so the value is in the join.
The figure turns the five buckets into a decision list. Read each row against the page you are checking.
- Crawled but uncited. The bot read a 200 and skipped you. Fix the content and structure, as in the fixes section.
- Blocked. The bot got a 403, a 429 or a timeout. Fix access at the edge, as in the setup section.
- Never crawled. The page is missing from the log. Fix discoverability with internal links and a clean sitemap. See the gap section.
- Cited and clicked. A live fetch bot hit, then a person arrived from chatgpt.com. Replicate that page across the whole topic cluster.
- Competitor wins. A rival page is cited for the prompt. Study it and rebuild yours to beat it.
Work through a short checklist each week. Read the logs: pull edge logs, bucket every hit by bot type, record each status code and match live fetch hits to referrals from chatgpt.com and perplexity.ai. Find the gaps: list pages crawled with a 200 that never get cited. List pages that return 403, 429 or timeouts. Add prompts whose target page is never crawled, and prompts where a rival is cited and you are absent. Fix and replicate: allow the citing bots, add a short answer up top, repair internal links and copy the pages that already earn live fetches and citations.
To carry that match through to signups, deals and payments, see AI search attribution in ThriveStack citedby.
Which answer engine optimization fixes win citations fastest?
Access fixes win fastest. If a search or live fetch bot cannot reach a page, no content work matters. Start in the log, clear every 403, 429 and timeout, then make each page easy to lift. Answer engine optimization starts with whether the bot can get in. Content tweaks come after that.
The figure gives the order. In step 1, clear any 403, 429 or timeout for the search and live fetch bots, then check robots.txt and your CDN rules for accidental blocks. Some hosts and CDNs block AI crawlers by default, so check yours. In step 2, speed up slow pages, because a timeout looks like a block from the bot’s side. Fix internal links and the sitemap so missing pages get crawled.
In step 3, make each page easy to lift. Put a short answer at the top, use question headings and add real numbers with sources. Skip keyword stuffing. It lowered AI visibility in the Princeton GEO study (2024).
The 404 figure is worth a look in your own log. Vercel and MERJ found that 34.82% of ChatGPT’s fetches and 34.16% of Claude’s hit a missing page, against 8.22% for Googlebot. Your log lists the missing URLs that bots ask for, so you can redirect the ones that matter to live pages.
The payoff is worth the work. Adobe data reported by Digital Commerce 360 shows AI referred visits to US retail sites converted 60% better than other traffic in July 2026. That covers retail only, so check conversion by source on your own site.
How do you pull Cloudflare Logpush data and verify AI bots?
Pull edge logs with Cloudflare Logpush, then confirm each bot against its vendor’s published IP list. Logpush streams the request log, including the user agent, the URL and the status code. Prefer edge logs, because a bot blocked at the edge never reaches your origin, so origin logs miss it.
The figure shows the four steps. Vercel has Log Drains and a plain server has access logs, so the same steps work without Cloudflare.
Verify every bot, because a user agent string is easy to fake. OpenAI publishes IP files for OAI-SearchBot, GPTBot and ChatGPT-User. Perplexity publishes files for PerplexityBot and Perplexity-User. Anthropic lists its crawler ranges at claude.com/crawling/bots.json. Match each hit’s IP to the list and treat a miss as an unknown or spoofed bot. Anthropic also warns that blocking by IP may stop a bot from reading robots.txt, so allow the listed ranges and manage the rest with rules.
GA4 and Search Console cannot run this check, so the raw log stays the source of truth.
Find the pages AI bots crawl but never cite
Connect your Cloudflare or server logs to ThriveStack citedby to uncover blocked AI bots, find uncited pages and track prompt-level citation gaps.
Start free AI log audit →AI Crawler Logs: FAQ
Can you track AI search visibility without server logs?
Not fully. Rank trackers and GA4 miss AI bots that skip JavaScript. Vercel and MERJ found that the bots from OpenAI, Anthropic, Perplexity, Meta and ByteDance do not render it. The server log is the only complete record of which AI crawlers reached your pages and what they got back.
Do AI crawlers respect robots.txt?
It depends on the bot. Anthropic says its bots honor robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page. A blanket block can still leak, and a careless block can still cost you citations.
What is the difference between retrieval and citation?
Retrieval means the AI pulled your page as a candidate source. Citation means it named your page in the answer. The gap between them is where most winnable opportunities hide.
What is a crawl to refer ratio?
It is the number of pages an operator crawls for every visit it sends back. Cloudflare Radar put Anthropic at 1,782 to 1 and Google at 4.79 to 1 in the 28 days to August 1, 2026. Cloudflare says native apps send no Referer header, so the ratios may overstate the gap.
Which AI crawler sends the most requests?
It changes with the window and the measure. In April 2026 ClaudeBot led GPTBot at 11.69% to 9.84% of AI bot requests, and in May GPTBot led at 11.48% to 9.73%, according to Cloudflare Radar data reported by Digital Applied. Check current data before you plan around a ranking.
Sources
- Vercel and MERJ, The Rise of the AI Crawler, Dec 17, 2024. Fetch counts, 404 rates, JavaScript rendering and the Ryan Siddle quote.
- Cloudflare, The crawl before the fall… of referrals, Jul 1, 2025. Crawl to refer ratio definition and the Referer header caveat.
- Cloudflare, A deeper look at AI crawlers: breaking down traffic by purpose and industry, Aug 28, 2025. Training at nearly 80% of AI bot crawling in July 2025.
- Cloudflare, Radar AI Insights. Source of the crawl to refer data for 2026.
- TechnologyChecker, Bot traffic statistics 2026. Cloudflare Radar crawl to refer ratios for the 28 days to Jun 22 and Aug 1, 2026 (read through this report).
- Digital Applied, AI crawler and bot traffic statistics 2026, Jun 12, 2026. Cloudflare Radar crawler shares for April and May 2026.
- OpenAI, Overview of OpenAI crawlers. GPTBot, OAI-SearchBot and ChatGPT-User roles, robots.txt and IP files.
- Anthropic, About Claude’s web crawler. ClaudeBot, Claude-SearchBot and Claude-User roles, robots.txt and IP ranges.
- Perplexity, Perplexity crawlers guide. PerplexityBot and Perplexity-User roles, IP files and firewall guidance.
- Peec AI, Why server logs are crucial for AI search strategy, Jun 30, 2026.
- Ahrefs, AI Search Overlap, Aug 11, 2025. Overlap of AI-cited URLs with Google’s top 10 across 15,000 prompts.
- Digital Commerce 360, Adobe AI referral traffic data, July 2026, Aug 19, 2026. Conversion of AI referred visits to US retail sites.
- Aggarwal et al., GEO: Generative Engine Optimization, Princeton University and others, 2024.
- Cloudflare, Logpush documentation, and Vercel, Log Drains documentation.
