ThriveStack citedbyWinning AI Search: AI search optimization courseAll coursesOverviewLive demoBook a demoStart free
Lesson 2.1 · Let AI crawlers in

AI crawler access:
Let search bots read your pages without JavaScript

An AI engine can only cite pages its crawler can reach and read. Allow the search bots, such as OAI-SearchBot, PerplexityBot and Claude-SearchBot, in robots.txt. Put key content in the server HTML, because the major AI crawlers do not run JavaScript. Then fix the 404s and redirects they hit.

0

major AI crawlers rendered JavaScript in Vercel’s study (vendor data)

Vercel, Dec 17, 2024
569M

GPTBot requests on Vercel’s network in one month

Vercel, Dec 17, 2024
34.82%

of ChatGPT crawler fetches hit 404 pages

Vercel, Dec 17, 2024
~20%

of the web is served by Cloudflare, which blocks AI crawlers by default on new domains

Cloudflare, Jul 1, 2025

Which AI bots should robots.txt allow?

Each company runs different bots for different jobs. Search and user bots decide whether you can be cited. Training bots only decide whether your pages train models. Allow the search bots, then make a deliberate choice on training.

ThriveStack citedbyCrawler policy

AI crawlers and what to do with each:
Search bots get in, training is your call

AI crawlers and what to do with each: Search bots get in, training is your call
BotJobRecommended rule
OpenAI logo OAI-SearchBotChatGPT searchAllow. Blocked sites do not appear in ChatGPT search answers.
OpenAI logo ChatGPT-UserVisits when a user asksAllow. robots.txt rules may not apply to it.
OpenAI logo GPTBotModel trainingYour call. Blocking opts you out of training only.
Perplexity logo PerplexityBotPerplexity searchAllow. Not used to train foundation models.
Anthropic logo Claude-SearchBotClaude search qualityAllow. Blocking may reduce visibility in Claude.
Anthropic logo ClaudeBotModel trainingYour call.
Google logo Google-ExtendedGemini training tokenYour call. Does not affect Google Search or AI Overviews.
Source: OpenAI, Perplexity, Anthropic and Google crawler docs, checked Nov 17, 2026thrivestack.ai/citedby · Nov 2026

What does Crawl Insights check?

Crawl Insights audits whether AI crawlers can reach and read your site, from robots.txt to server-side rendering and page speed. A good sign is every critical row passing. A warning sign is a Fail on robots.txt or rendering, which blocks every later stage.

ThriveStack citedby Crawl Insights with 28 crawl-readiness actions, priority and audit result1234
  1. 1Progress on 28 crawl-readiness actions.
  2. 2Two views: citation opportunities and configuration issues.
  3. 3Allow all major AI crawlers in robots.txt, marked critical.
  4. 4Pass, Warn or Fail from the latest audit.
Crawl Insights: can AI crawlers reach and read your website? ThriveStack citedby demo workspace. Sample data.

What else blocks AI crawlers?

Technical checklist for AI crawlers
  • Server-render key content: crawlers fetch JavaScript files but do not run them
  • Check Cloudflare or your CDN: new Cloudflare domains block AI crawlers by default
  • Fix 404s and redirect chains that AI bots hit, found in your server logs
  • Submit an XML sitemap with accurate lastmod dates to Google and Bing, and use IndexNow
  • Treat llms.txt as optional: Google says no AI system currently uses it

Bing says it reads lastmod as a key freshness signal and ignores changefreq and priority. Google says no special AI files or markup are needed for AI Overviews or AI Mode.

Research for this lesson.

Our own studies behind this lesson. Each one has the key charts and numbers, and a link to the full article.

How ThriveStack citedby helps

ThriveStack citedby audits crawl readiness, reads your server logs for AI bot visits and errors, and lists each fix with an owner and priority.

Audit my crawler access →

AI crawler access, answered.

Should I block GPTBot?

Only if you do not want your content used for training. Blocking GPTBot does not remove you from ChatGPT search, which uses OAI-SearchBot.

Do AI crawlers run JavaScript?

Vercel found none of the major AI crawlers rendered JavaScript in its study. Put the content you want cited in the server HTML.

Do I need llms.txt?

It is optional. Google’s John Mueller said no AI system currently uses it, and Google says no special AI files are needed.

Does Google-Extended affect AI Overviews?

No. Google says Google-Extended does not affect inclusion in Google Search. AI Overviews follow normal Googlebot rules.

All 10 questions this page answers.

These questions come from a query fan-out of "how to win AI search visibility". Each one has a short answer here, and the lesson above goes deeper.

How to allow OAI-SearchBotImplementation

In robots.txt add a "User-agent: OAI-SearchBot" group with "Allow: /", and check that your CDN or firewall does not block it. OpenAI says robots.txt changes take about 24 hours to apply. Then confirm in server logs that it gets status 200.

How to allow PerplexityBotImplementation

Add a "User-agent: PerplexityBot" group with "Allow: /" and check your CDN and firewall rules. Perplexity says changes can take up to 24 hours. Confirm in server logs that PerplexityBot gets status 200 on the pages you want cited.

AI crawler robots.txt best practicesCommercial

Allow the search and user bots, decide on training bots on purpose, avoid blanket Disallow rules, and keep the file at your domain root. Recheck after any CDN change: Cloudflare has blocked AI crawlers by default on new domains since Jul 1, 2025.

GPTBot vs OAI-SearchBotCommercial

GPTBot collects content that may train OpenAI models. Blocking it opts you out of training. OAI-SearchBot powers ChatGPT search, and blocking it keeps you out of search answers. You can block GPTBot and still allow OAI-SearchBot.

AI bot access blocked by CloudflareTroubleshooting

Cloudflare blocks AI crawlers by default on new domains since Jul 1, 2025, and it serves about 20% of the web. Review the AI crawler and bot settings in your Cloudflare dashboard, allow the search bots you want, then confirm in logs that they get status 200.

How to test AI crawler accessImplementation

Read robots.txt for each bot, request a page with the bot’s user agent to see the status code, and check server logs for real visits. A 200 means the bot got the page. A 403, 404 or redirect loop means it did not.

AI crawler IP verificationInformational

Anyone can fake a user agent, so check that the request comes from the IP ranges the bot’s owner publishes, or confirm it with a reverse DNS lookup. ThriveStack citedby counts only IP-verified bot fetches when it matches crawls to citations.

ChatGPT-User vs OAI-SearchBotCommercial

OAI-SearchBot crawls ahead of time for ChatGPT search. ChatGPT-User visits a page when a user’s question needs it, and OpenAI says robots.txt rules may not apply to it. A ChatGPT-User hit in your logs means a live user question.

How to check AI bot server logsImplementation

Filter your logs by AI user agents such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot. Look at pages, status codes and dates. Seer suggests comparing their status codes with Googlebot’s and fixing redirects and errors. ThriveStack citedby reads logs from Vercel, WordPress or a Cloudflare Worker.

Does blocking AI training bots affect AI search?Informational

Mostly no. Blocking GPTBot opts out of OpenAI training, not ChatGPT search. Google says Google-Extended does not affect inclusion in Google Search. Blocking ClaudeBot stops Claude training, while blocking Claude-SearchBot or Claude-User may cut Claude search visibility.

Sources for this lesson

  1. OpenAI: Overview of OpenAI crawlers Checked Nov 17, 2026
  2. Perplexity: Perplexity crawlers Checked Nov 17, 2026
  3. Anthropic: Does Anthropic crawl data from the web Updated Oct 30, 2026
  4. Google Search Central: Google common crawlers Google-Extended
  5. Vercel: The rise of the AI crawler (Dec 17, 2024) Vendor claim, Vercel network data
  6. Cloudflare: Default AI crawler blocking (Jul 1, 2025)
  7. Bing Webmaster Blog: Sitemaps in AI-powered search (Jul 31, 2025)
  8. Search Engine Roundtable: John Mueller on llms.txt (Jun 18, 2025)
  9. Google Search Central: AI features and your website Checked Nov 17, 2026

Figures were checked against each page on Nov 17, 2026. Vendor studies are labelled as vendor claims.

Go deeper
MapHover to open