Analysing AI Crawler Logs:
How to Find and Fix AI Citation Gaps

AI crawler logs are server records of each request an AI bot makes to your site: its name, the page and the status code it got. They show which bots reached you, which were blocked and which never came. Matching them to tracked prompts finds the citation gaps to fix first.

ThriveStack citedby Five fixes: what to do for each AI crawler log pattern Cross what the log says the bot did with what the tracker says the prompt returned IF THIS IS YOU START WITH 1 Bot read a 200,you were never named Crawled but uncited Fix content and structure 2 Bot got a 403, 429or a timeout Blocked Fix access at the edge 3 The page is missingfrom the log Never crawled Fix internal links, sitemap 4 Live fetch hit, then avisit from chatgpt.com Cited and clicked Replicate across the cluster 5 A rival is citedand you are absent Competitor wins Rebuild the page to beat it Source: ThriveStack citedby framework. Orange marks the fastest win
Read on

Four numbers frame this guide. Vercel and MERJ counted 569 million GPTBot fetches in one month and found that none ran JavaScript, so analytics never saw them (section 1). Anthropic crawled 1,782 pages for every referral it sent in mid 2026, against 4.79 for Google (section 4). About 35% of ChatGPT’s fetches hit a missing page, which a log shows and GA4 does not (section 7). And AI referred visits to US retailers converted 60% better in July 2026, which is why ThriveStack citedby helps you close the gaps (section 7).

569M
GPTBot fetches in one month on Vercel, none ran JavaScript
1,782:1
pages Anthropic crawled per referral, 28 days to Aug 1, 2026
34.8%
of ChatGPT bot fetches hit a missing page (404)
60%
better conversion for AI referred visits, US retail, Jul 2026

Our research with Vercel highlights that AI crawlers, while rapidly scaling, continue to face significant challenges in handling JavaScript and efficiently crawling content.

Ryan Siddle, Managing DirectorMERJ
Vercel, The Rise of the AI Crawler, Dec 17, 2024
01 · The ground truth

Why does AI search visibility depend on server logs?

AI answers change on every run, so there is no rank to check, and analytics never sees the bots. Your server log is the one record of what an AI engine did with your site: which bot came, which page it asked for and which status code it got back. That makes AI search visibility something you can check.

ThriveStack citedby Why GA4 misses AI bots and the server log does not What each tool records when an AI crawler asks for one of your pages AI botasks for a page GA4 tagruns only if JavaScript runs Nothing recordedThe tag never fires Server or edge logrecords the request itself Bot, page, status, timeEach request is kept AI bots read the raw HTML and leave, so only the log sees them Source: Vercel and MERJ, Dec 2024. OpenAI and Anthropic bots do not run JavaScript
Dashed means the step depends on JavaScript. Orange marks what the log keeps.

The figure shows why the two tools disagree. GA4 and most marketing analytics count a visit only when a script runs in a browser. AI crawlers read the raw HTML and leave. Vercel and MERJ measured this on Vercel’s network in late 2024. In one month they saw 569 million fetches from OpenAI’s GPTBot and 370 million from Claude. They found that the bots from OpenAI, Anthropic, Perplexity, Meta and ByteDance do not render JavaScript. Google’s Gemini crawler and AppleBot did.

Because the tag never fires, the visit never reaches your dashboard. Tag managers, heatmaps and most attribution tools share the blind spot. A rank tracker has its own limit. It was built for ten blue links, and an AI answer has none.

The server log has none of these limits, because it records the request itself. The log is the receipt. It tells you whether an AI engine reached a page and what it got when it did. Peec AI makes the same case: logs show whether key bots are blocked or show little interest in a domain.

Read next: What GA4 can and cannot see about AI traffic.

02 · The shortcut

How does ThriveStack citedby show what GPTBot did with your pages?

ThriveStack citedby reads your server logs, runs your tracked prompts through the AI engines and joins the two on each URL. You see which pages a bot fetched, which pages an answer cited and where the two do not match. Those mismatches are your citation gaps, ranked by what to fix first.

ThriveStack citedby Server logs plus prompt tests reveal your citation gaps ThriveStack citedby joins three layers on each URL. The four log sources shown are examples YOUR LOG SOURCES Cloudflare Vercel WordPress AWS YOUR PROMPT PANEL Prompts buyers askChatGPT, Perplexity,Gemini and more LAYER 1 Server logs Which AI bot fetched whichURL, and the status code it got LAYER 2 Synthetic prompts What each AI engine answers andwhich URLs it cites in the answer JOIN ON URL Cited Fetched and named in theanswer. Copy what works. Citation gap Blocked, skipped or neverfetched. Fix these first. One view per page: what the bot did, what the AI said and what to fix next. Source: ThriveStack citedby framework. Logo marks belong to their owners
Orange marks the gaps. Green marks pages that earned a citation.

The figure shows the flow. Layer 1 is your own log, from a source such as Cloudflare, Vercel, WordPress or AWS. Layer 2 is synthetic prompt analysis: the questions your buyers ask, asked of ChatGPT, Perplexity and Gemini on a schedule, with the cited URLs recorded. The join happens on the page address, so each row tells one story.

A cited page shows a successful fetch and a citation. A gap shows a block, a skip or no fetch at all. That is the same reading this guide teaches by hand, done for every page at once. It is a shortcut for the method below, and you can follow the method without it.

03 · The three crawlers

What are the three types of AI crawlers in your logs?

There are three types: training bots, search bots and live fetch bots. Each does a different job and leaves a different mark in your log. Training bots feed the model. Search bots build the index that answers draw from. Live fetch bots grab one page for one prompt. Function matters more than the company name.

ThriveStack citedby Three AI crawler types: training, search and live fetch Bot names that appear in your server log, grouped by the job each one does TRAINING GPTBotClaudeBot Feeds the model.No direct citations. SEARCH INDEX OAI-SearchBotClaude-SearchBotPerplexityBot Builds the indexanswers draw from. LIVE FETCH ChatGPT-UserClaude-UserPerplexity-User Fetches one pagefor one prompt. Cited EARLIER WHEN THE PROMPT FIRES Source: OpenAI, Anthropic and Perplexity bot documentation, 2026
Orange marks the bot class that lines up with a live citation.

The figure sorts the bots you will see in a log. OpenAI, Anthropic and Perplexity each publish their bot lists, so you can group hits by name. OpenAI describes GPTBot as its training crawler and OAI-SearchBot as the one that surfaces sites in ChatGPT search. ChatGPT-User handles actions a person starts. Anthropic runs ClaudeBot for training, Claude-SearchBot for search quality and Claude-User for user questions. Perplexity splits PerplexityBot, which feeds its search results, from Perplexity-User, which handles user actions.

The split decides what a block costs you. Blocking a training bot keeps your pages out of model training. It leaves you in the answers. OpenAI says sites that opt out of OAI-SearchBot will not appear in ChatGPT search answers, and Anthropic says blocking Claude-SearchBot or Claude-User may reduce your visibility. So the wrong block quietly removes you from answers.

Check robots.txt, and check your CDN and firewall too. A bot rule at that layer can return a 403 that robots.txt never shows. Live fetch bots follow different rules. OpenAI says robots.txt rules “may not apply” to ChatGPT-User because a person asked for the page, and Perplexity says Perplexity-User generally ignores them. Anthropic says its bots honor robots.txt. A starter file that lets the citing bots in looks like this:

# Allow the bots that can cite you
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /
ThriveStack citedby Searches for AI crawler names, last 12 months Monthly US Google searches. OAI-SearchBot rose 2.8 times from Oct 2025 to Sep 2026 0 200 400 600 Oct '25 Dec Feb Apr Jun Aug 170 AI crawler 590/mo OAI-SearchBot 480/mo GPTBot 260/mo PerplexityBot 140/mo Source: DataForSEO Google Ads search volume, US, Oct 2026Sep '26
Searches for OAI-SearchBot are rising fastest. ClaudeBot peaked at 33,100 in Feb 2026 and fell to 880 by Sep, so it sits off this scale.

Read next: Is blocking AI bots always a bad idea?

04 · What logs show

What does log file analysis show that analytics cannot?

It shows every AI bot hit and the status code each one got, which GA4 and Search Console never record. A 200 means the bot read the page. A 403 or a timeout means it could not. No amount of good content fixes a block, so the status code is the first thing to read.

ThriveStack citedby AI bots crawl far more pages than they send visitors back Pages a bot reads for each 1 visit it sends to a site, 28 days to Aug 1, 2026 Anthropic 1,782 pages Perplexity 303 pages OpenAI 233 pages Google 4.79 pages Read it like this: Anthropic reads about 1,782 pages to send 1 visitor. Google reads about 5. Source: Cloudflare Radar, read through TechnologyChecker, Aug 2026
Orange marks Anthropic, the widest gap. Bars use a plain scale, so Google’s 5 shows as a sliver.

The log also shows the gap between crawling and visits. Cloudflare tracks pages crawled per referral, the crawl to refer ratio. In the 28 days to June 22, 2026, Anthropic crawled 4,580 pages for every referral it sent, OpenAI 848 and Google 5. By the 28 days to August 1 the figures were 1,782, 233 and 4.79. Perplexity went the other way, from 186 to 303. The ratios move between windows, so read them as a range. Cloudflare also warns they may overstate the gap, because native apps send no Referer header.

Training still dominates what bots fetch. In July 2025 Cloudflare put training at nearly 80% of AI bot crawling. Heavy crawling therefore says little about citations. It measures data collection. A citation needs an open path to your page and content that an engine chooses to use. Our fact-check on whether crawler logs guarantee citations covers this in more detail.

ThriveStack citedby Reading AI bot log lines: a block, a skip and a citation Three example log lines and what each one tells you. The lines are made up 403 GET /compare/ai-visibility | Perplexity-User/1.0 A SILENT BLOCK The edge turned the bot away. You dropped out of the answer and analytics saw nothing. 200 GET /blog/ai-search-guide | OAI-SearchBot/1.0 CRAWLED, UNCITED The bot read the page and still chose a rival. That points to content rather than access. 200 GET /vs/example-rival | ChatGPT-User/2.0 CONFIRMED CITATION A person then arrives from chatgpt.com, so this page made it into the answer. Example data. Paths and prompts are made up
Red marks a block, amber a skip and green a confirmed citation.
  1. The silent block. Perplexity-User asked for the page and the edge returned a 403. Fix access first, since no other work helps. See the setup section.
  2. Crawled, uncited. The 200 rules out access, and the bot still chose a rival. That points to content and structure. See the fixes section.
  3. Confirmed citation. A live fetch, then a person arriving from chatgpt.com moments later. Find more pages like it. See the method section.
05 · The gap

What is citation gap analysis in AI search?

Citation gap analysis finds the prompts where competitors get cited and you do not. It splits two steps that look alike. Retrieval means the AI pulled your page as a candidate source. Citation means the AI named you in the answer. A page can be retrieved often and cited rarely.

ThriveStack citedby Where pages leak between your sitemap and an AI citation How 1,000 sitemap pages flow to Google and to AI bots. Made-up numbers for one example site SitemapGoogle crawled 800Not crawled 200AI bot hits 450No AI bot hits 550Google indexed 620Not indexed 180Cited by AI 180Not cited 270SEARCH CONSOLE + SERVER LOGSOUTCOMEGoogle funnelAI bot hitsDropped offMade-up numbers, 1,000 sitemap pages Illustrative numbers. Framework from ThriveStack citedby
Band width shows page count. Red bands are pages that dropped off. Orange bands follow the AI path to a citation.

The figure follows 1,000 sitemap pages down two paths. One goes through Google crawling and indexing, the other through AI bot hits and citations. The numbers are made up to show the shape. Search Console shows indexing, your logs show which AI bots live crawled a page, and a prompt tracker shows which pages got named. Each leak has its own fix: tune for indexing, for AI readability or for the prompts.

Peec AI describes the same gap. A page that bots request often and prompts never name is being considered but not selected. That is the richest place to win, because the content is already close to good enough. AI answers change from run to run, so read citation trends over weeks rather than one day. For the wider picture, see our research on AI citation sources in 2026 and the AI visibility gap report.

ThriveStack citedby Most AI-cited pages do not rank in Google's top 10 Share of AI-cited URLs that also rank in Google's top 10. 15,000 long-tail prompts Perplexity 28.6% ChatGPT, in-text links 8.0% ChatGPT, references 6.1% Average Ahrefs reports 11.9% About one cited page in eight also ranks in the top 10 Source: Ahrefs, AI search overlap study, Aug 2025
Orange marks the average. Ahrefs reports Perplexity as the one engine near 30%.

Google position does not tell you who gets cited. Ahrefs checked 15,000 long-tail prompts. Pages that Perplexity cited overlapped Google’s top 10 at 28.6%, and ChatGPT’s in-text citations at 8.0%. Across the engines Ahrefs reports, the average was about 11.9%. You cannot read your AI visibility off your Google rank.

Sitemap + AI indexing

See which of your pages AI can actually cite

Connect Google Search Console to ThriveStack citedby and the whole pipeline shows up in one view: every sitemap page, Google crawled vs indexed, the pages AI bots fetched in your server logs, and the citations they earned.

See the sample pipeline →Sample data, no signup. Connect Search Console when you are ready.
SitemapGoogle crawledNot crawledAI bot hitsNo AI bot hitsGoogle indexedNot indexedCited by AINot citedSEARCH CONSOLE + SERVER LOGSOUTCOMEGoogle funnelAI bot hitsDropped offSample data. Your numbers appear once Search Console is connected.
06 · The method

How do you match AI crawler logs to AI citation gaps?

Cross two lists: what your logs say each bot did, and what your tracker says each prompt returned. The overlap sorts every page into one of five buckets, and each bucket has one fix. The logs and the tracker are each half the picture, so the value is in the join.

ThriveStack citedby Five fixes: what to do for each AI crawler log pattern Cross what the log says the bot did with what the tracker says the prompt returned IF THIS IS YOU START WITH 1 Bot read a 200,you were never named Crawled but uncited Fix content and structure 2 Bot got a 403, 429or a timeout Blocked Fix access at the edge 3 The page is missingfrom the log Never crawled Fix internal links, sitemap 4 Live fetch hit, then avisit from chatgpt.com Cited and clicked Replicate across the cluster 5 A rival is citedand you are absent Competitor wins Rebuild the page to beat it Source: ThriveStack citedby framework. Orange marks the fastest win
The figure also sits at the top of this page. Orange marks the fastest win.

The figure turns the five buckets into a decision list. Read each row against the page you are checking.

  1. Crawled but uncited. The bot read a 200 and skipped you. Fix the content and structure, as in the fixes section.
  2. Blocked. The bot got a 403, a 429 or a timeout. Fix access at the edge, as in the setup section.
  3. Never crawled. The page is missing from the log. Fix discoverability with internal links and a clean sitemap. See the gap section.
  4. Cited and clicked. A live fetch bot hit, then a person arrived from chatgpt.com. Replicate that page across the whole topic cluster.
  5. Competitor wins. A rival page is cited for the prompt. Study it and rebuild yours to beat it.

Work through a short checklist each week. Read the logs: pull edge logs, bucket every hit by bot type, record each status code and match live fetch hits to referrals from chatgpt.com and perplexity.ai. Find the gaps: list pages crawled with a 200 that never get cited. List pages that return 403, 429 or timeouts. Add prompts whose target page is never crawled, and prompts where a rival is cited and you are absent. Fix and replicate: allow the citing bots, add a short answer up top, repair internal links and copy the pages that already earn live fetches and citations.

To carry that match through to signups, deals and payments, see AI search attribution in ThriveStack citedby.

07 · The fixes

Which answer engine optimization fixes win citations fastest?

Access fixes win fastest. If a search or live fetch bot cannot reach a page, no content work matters. Start in the log, clear every 403, 429 and timeout, then make each page easy to lift. Answer engine optimization starts with whether the bot can get in. Content tweaks come after that.

ThriveStack citedby AI citation fixes in order: access, then speed, then content A suggested order for pages that AI bots can already find in your log 1 STEP 1 Open access Clear every 403, 429 and timeout Allow search and live fetch bots Check CDN and WAF rules 2 STEP 2 Speed up and link Fix slow pages that time out Redirect missing URLs bots keep asking for Fix internal links and the sitemap 3 STEP 3 Make it liftable Short answer at the top Question headings Numbers with sources Source: ThriveStack citedby recommended order. No study sits behind it
Orange marks the step that pays back first.

The figure gives the order. In step 1, clear any 403, 429 or timeout for the search and live fetch bots, then check robots.txt and your CDN rules for accidental blocks. Some hosts and CDNs block AI crawlers by default, so check yours. In step 2, speed up slow pages, because a timeout looks like a block from the bot’s side. Fix internal links and the sitemap so missing pages get crawled.

In step 3, make each page easy to lift. Put a short answer at the top, use question headings and add real numbers with sources. Skip keyword stuffing. It lowered AI visibility in the Princeton GEO study (2024).

ThriveStack citedby AI bots hit missing pages four times as often as Google Share of fetches that returned a 404. Vercel network, one month in late 2024 ChatGPT (OpenAI bots) 34.8% Claude (Anthropic bots) 34.2% Googlebot 8.2% Shown for comparison Source: Vercel and MERJ, The Rise of the AI Crawler, Dec 2024
Orange marks the two AI bot families. Googlebot is the baseline.

The 404 figure is worth a look in your own log. Vercel and MERJ found that 34.82% of ChatGPT’s fetches and 34.16% of Claude’s hit a missing page, against 8.22% for Googlebot. Your log lists the missing URLs that bots ask for, so you can redirect the ones that matter to live pages.

The payoff is worth the work. Adobe data reported by Digital Commerce 360 shows AI referred visits to US retail sites converted 60% better than other traffic in July 2026. That covers retail only, so check conversion by source on your own site.

08 · The setup

How do you pull Cloudflare Logpush data and verify AI bots?

Pull edge logs with Cloudflare Logpush, then confirm each bot against its vendor’s published IP list. Logpush streams the request log, including the user agent, the URL and the status code. Prefer edge logs, because a bot blocked at the edge never reaches your origin, so origin logs miss it.

ThriveStack citedby From raw edge log to fix list in four steps How to pull, group, verify and join AI crawler data 1 Pull edge logs Cloudflare Logpush,Vercel Log Drainsor server logs 2 Bucket by bot Group hits astraining, searchor live fetch.Keep status codes. 3 Verify the bot Check each IPagainst the vendor'spublished list.Drop spoofed hits. 4 Join prompts Match live fetchhits to promptsand chatgpt.comreferrals. Result: every page sorted into one of five fixes Sources: Cloudflare, OpenAI, Anthropic and Perplexity documentation
Step 3 is the one teams skip. A fake user agent can pass for a real bot.

The figure shows the four steps. Vercel has Log Drains and a plain server has access logs, so the same steps work without Cloudflare.

Verify every bot, because a user agent string is easy to fake. OpenAI publishes IP files for OAI-SearchBot, GPTBot and ChatGPT-User. Perplexity publishes files for PerplexityBot and Perplexity-User. Anthropic lists its crawler ranges at claude.com/crawling/bots.json. Match each hit’s IP to the list and treat a miss as an unknown or spoofed bot. Anthropic also warns that blocking by IP may stop a bot from reading robots.txt, so allow the listed ranges and manage the rest with rules.

GA4 and Search Console cannot run this check, so the raw log stays the source of truth.

Find the pages AI bots crawl but never cite

Connect your Cloudflare or server logs to ThriveStack citedby to uncover blocked AI bots, find uncited pages and track prompt-level citation gaps.

Start free AI log audit →
Frequently asked questions

AI Crawler Logs: FAQ

Can you track AI search visibility without server logs?

Not fully. Rank trackers and GA4 miss AI bots that skip JavaScript. Vercel and MERJ found that the bots from OpenAI, Anthropic, Perplexity, Meta and ByteDance do not render it. The server log is the only complete record of which AI crawlers reached your pages and what they got back.

Do AI crawlers respect robots.txt?

It depends on the bot. Anthropic says its bots honor robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page. A blanket block can still leak, and a careless block can still cost you citations.

What is the difference between retrieval and citation?

Retrieval means the AI pulled your page as a candidate source. Citation means it named your page in the answer. The gap between them is where most winnable opportunities hide.

What is a crawl to refer ratio?

It is the number of pages an operator crawls for every visit it sends back. Cloudflare Radar put Anthropic at 1,782 to 1 and Google at 4.79 to 1 in the 28 days to August 1, 2026. Cloudflare says native apps send no Referer header, so the ratios may overstate the gap.

Which AI crawler sends the most requests?

It changes with the window and the measure. In April 2026 ClaudeBot led GPTBot at 11.69% to 9.84% of AI bot requests, and in May GPTBot led at 11.48% to 9.73%, according to Cloudflare Radar data reported by Digital Applied. Check current data before you plan around a ranking.

Sources

  1. Vercel and MERJ, The Rise of the AI Crawler, Dec 17, 2024. Fetch counts, 404 rates, JavaScript rendering and the Ryan Siddle quote.
  2. Cloudflare, The crawl before the fall… of referrals, Jul 1, 2025. Crawl to refer ratio definition and the Referer header caveat.
  3. Cloudflare, A deeper look at AI crawlers: breaking down traffic by purpose and industry, Aug 28, 2025. Training at nearly 80% of AI bot crawling in July 2025.
  4. Cloudflare, Radar AI Insights. Source of the crawl to refer data for 2026.
  5. TechnologyChecker, Bot traffic statistics 2026. Cloudflare Radar crawl to refer ratios for the 28 days to Jun 22 and Aug 1, 2026 (read through this report).
  6. Digital Applied, AI crawler and bot traffic statistics 2026, Jun 12, 2026. Cloudflare Radar crawler shares for April and May 2026.
  7. OpenAI, Overview of OpenAI crawlers. GPTBot, OAI-SearchBot and ChatGPT-User roles, robots.txt and IP files.
  8. Anthropic, About Claude’s web crawler. ClaudeBot, Claude-SearchBot and Claude-User roles, robots.txt and IP ranges.
  9. Perplexity, Perplexity crawlers guide. PerplexityBot and Perplexity-User roles, IP files and firewall guidance.
  10. Peec AI, Why server logs are crucial for AI search strategy, Jun 30, 2026.
  11. Ahrefs, AI Search Overlap, Aug 11, 2025. Overlap of AI-cited URLs with Google’s top 10 across 15,000 prompts.
  12. Digital Commerce 360, Adobe AI referral traffic data, July 2026, Aug 19, 2026. Conversion of AI referred visits to US retail sites.
  13. Aggarwal et al., GEO: Generative Engine Optimization, Princeton University and others, 2024.
  14. Cloudflare, Logpush documentation, and Vercel, Log Drains documentation.