ThriveStack citedbyWinning AI Search: AI search optimization courseAll coursesOverviewLive demoBook a demoStart free
Hub 2 · 2 lessons and 2 research pages

Become eligible:
Crawling, indexing and rendering for AI

An engine can only cite a page its crawler can reach, read and index. Let the AI search bots in, put content in the server HTML, keep the sitemap accurate and get indexed. Nothing later in the course counts until this works.

ThriveStack citedby

Becoming eligible for AI search:
Five gates every page must pass

1Crawler allowedrobots.txt and CDN let AI search bots in2Server-renderedText is in the HTML, no JavaScript needed3Status 200No 404s or redirect chains in the logs4IndexedIn Google and Bing, eligible for a snippet5RetrievableReady to be found and cited
Becoming eligible for AI search: five gates every page must pass
How to read it

What must be true before an AI engine can cite a page?

Eligibility is a chain of gates. A page that fails one gate is invisible to everything after it, so check them in order and fix the first one that fails.

  1. Each box is one gate, in the order a crawler meets it.
  2. The arrows show the dependency: a later gate never helps if an earlier one fails.
  3. The last box is the goal: a page an engine can retrieve.

Good sign: AI search bots get status 200 on your key pages in server logs.

Warning sign: new pages stay unindexed, or bots see an empty page without JavaScript.

0

major AI crawlers rendered JavaScript in Vercel’s study (vendor data)

Vercel, Dec 17, 2024
569M

GPTBot requests on Vercel’s network in one month

Vercel, Dec 17, 2024
34.82%

of ChatGPT crawler fetches hit 404 pages

Vercel, Dec 17, 2024
~20%

of the web is served by Cloudflare, which blocks AI crawlers by default on new domains

Cloudflare, Jul 1, 2025

Lessons in Become eligible.

  1. Lesson 2.1

    Let AI crawlers in

    AI crawlers must reach and read your pages before you can be cited. Allow the right bots in robots.txt, render content on the server and fix crawl errors.
  2. Lesson 2.2

    Get indexed

    Indexing and technical SEO for AI search: sitemaps with lastmod, IndexNow, server rendering, canonicals and noindex checks that decide AI eligibility.
  3. Research 2.3

    Improve page speed

    Pages with first contentful paint under 0.4s averaged 6.7 ChatGPT citations, against 2.1 over 1.13s. See our speed fixes and test plan.
  4. Research 2.4

    Rank in AI Overviews

    In eight weeks we fixed ten issues, raised indexing from 42 to 192 of 199 pages and grew pages cited by AI from 2 to more than 100.

See this hub in ThriveStack citedby.

How ThriveStack citedby helps

ThriveStack citedby audits crawl readiness, reads your server logs for AI bot visits and errors, and lists each fix with an owner and priority.

Audit my crawler access →

MapHover to open