AI crawler server logs:
what they show that analytics cannot
AI crawlers do not run JavaScript, so your analytics tag never fires for them. Server logs are the only place their visits show up. Read them by bot type, folder and URL to see what engines can read, then compare with human visits to see what they send people to.
of AI crawling was for model training
Cloudflare, Aug 29, 2025crawl to refer ratio for Perplexity, Jul 2025
Cloudflare, Aug 29, 2025crawl to refer ratio for OpenAI, Jul 2025
Cloudflare, Aug 29, 2025crawl to refer ratio for Anthropic, Jul 2025
Cloudflare, Aug 29, 2025
Anthropic read 38,066 pages for each visit it sent back in July 2025, OpenAI 1,091 and Perplexity 195
Crawl to refer ratio: pages an AI company crawled for each referred visit, Jul 2025.
Source: Cloudflare, Aug 29, 2025 (Jul 2025 data). Training made up about 80% of AI crawling.
How to read it. A high ratio means the company reads a lot and sends little traffic back. A low ratio means each page read is more likely to earn a visit.
In ThriveStack citedby Crawler hits by bot, folder and URL sit beside human visits, so you see which pages engines read and which they send people to. See it for your site →
AI crawlers do not run JavaScript, so only server logs see them.
An analytics tag is a script. A crawler fetches the page and skips the script, so GA4 records nothing. The web server still writes a line for every request.
Three kinds of bot do three different jobs.
Training bots collect pages to build models. Search bots find pages to cite in answers. User query bots fetch a page when a person asks. Cloudflare found training made up about 80% of AI crawling.
Three kinds of AI bot, and the job each one does
| Bot type | Job | Examples |
|---|---|---|
| Training | Collects pages to build models | GPTBot, ClaudeBot |
| Search | Finds pages to cite in answers | OAI-SearchBot, PerplexityBot |
| User query | Fetches a page for a person's request | ChatGPT-User |
Read crawls by folder and URL to find what engines read.
The demo workspace below shows 12,708 bot visits in seven days, split by folder, URL and bot. Sample data.
- Which bots visit most
- Which folders and URLs they read
- Failure rate and status codes
- Which important pages are not crawled at all
Compare crawls with referred visits to find weak pages.
A page crawled often and referred rarely usually has a weak or buried answer. Anthropic read 38,066 pages for each visit it sent back in July 2025, so crawl volume is not the same as traffic.
pages Perplexity read per visit referred, Jul 2025
Cloudflare, Aug 29, 2025pages OpenAI read per visit referred
Cloudflare, Aug 29, 2025pages Anthropic read per visit referred
Cloudflare, Aug 29, 2025ThriveStack citedby runs your prompt panel across 10+ AI engines, reads server-log crawler hits, matches human visits to signups and reports AI-sourced and AI-influenced pipeline.
AI crawler server logs, answered.
Why do I need server logs for AI crawlers?
AI crawlers do not run your analytics script, so only server logs record their visits.
What is a crawl to refer ratio?
The number of pages a company crawls for each visit it sends back to sites.
Which bots should I watch?
Training bots, search bots and user query bots, read separately.
Sources for this lesson
- Cloudflare: The crawlers that never click (Aug 29, 2025) Crawl to refer ratios, Jul 2025
Figures were checked against each page in Nov 2026. Vendor studies are marked by their sample. Charts marked sample data are illustrations.
All 10 questions this lesson answers
AI crawler server log analytics
Your server log records every request, including bots that never run JavaScript. Filter by user agent, such as GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot and ClaudeBot, to see which pages AI systems fetch and how often. Logs show fetches, not human visits. ThriveStack citedby reads crawler data from Cloudflare, Vercel and WordPress and shows it next to human visits from GA.
GPTBot vs ChatGPT-User traffic
GPTBot gathers content that may be used to train models. ChatGPT-User fetches a page when a person asks ChatGPT something that needs it, and OAI-SearchBot builds the search index. A ChatGPT-User hit is a sign of live demand. None of the three is a human visit.
PerplexityBot vs human referral visits
PerplexityBot crawls pages to build its index. A human referral is a person who clicks a Perplexity link, arrives with a perplexity.ai referrer and runs your analytics script. Crawler hits live in your logs, and referral visits live in your analytics, so compare them page by page.
AI crawler activity vs user conversions
Crawlers do not sign up or buy, so crawler hits are not conversions. Join the two data sets by page. Pages that AI bots fetch often and that also get human AI referrals are the pages to protect and improve. Treat crawl volume as an early signal, not as proof of revenue.
How to analyze AI bot requests
Export the raw logs for a set period. Match user agents to known AI bots, then group hits by bot, URL and status code. Check that the IP address matches the ranges the vendor publishes, because user agents can be faked. Look for 4xx and 5xx errors, since a blocked or broken page cannot be cited.
Cloudflare AI bot traffic analytics
Cloudflare AI Crawl Control shows which AI crawlers hit your site and which paths they request, with no log export needed. It is the fastest way to start if your site already runs on Cloudflare. It still counts bots only, so pair it with human visits from your analytics tool.
AI crawler log attribution limitations
Logs cannot tell you who a person is, whether a page was cited, or whether anyone converted. A CDN cache can hide hits from your origin server. User agents can be spoofed, and some AI browsers look exactly like Chrome. Use logs to see access, and use other data to see impact.
AI citation discovery through server logs
A ChatGPT-User or Perplexity-User hit on a URL tells you the page was fetched to answer a prompt. It does not tell you the prompt, the answer or whether you were cited. Confirm citations by tracking the prompts directly, as ThriveStack citedby does for tracked prompts across AI engines.
How to correlate AI bot visits and referrals
Put both data sets on the same time zone and the same URL format. Then line up bot fetches and AI referral visits by page and by day. A page fetched by a bot on Monday and visited from chatgpt.com on Tuesday is a pattern worth noting, not proof of cause.
AI bot visibility vs revenue attribution
Visibility is what AI systems can read and cite. Revenue is what people do after they arrive, tracked through to a closed deal. Report them side by side and never convert one into the other. ThriveStack Revenue Intelligence links website visitors with CRM and billing records, which is the revenue side of this view.
Sources, checked November 2026: OpenAI crawler documentation · Perplexity crawler documentation · Anthropic crawler documentation · Cloudflare AI Crawl Control · Matomo AI reports guide.