Become eligible:
Crawling, indexing and rendering for AI
An engine can only cite a page its crawler can reach, read and index. Let the AI search bots in, put content in the server HTML, keep the sitemap accurate and get indexed. Nothing later in the course counts until this works.
Becoming eligible for AI search:
Five gates every page must pass
What must be true before an AI engine can cite a page?
Eligibility is a chain of gates. A page that fails one gate is invisible to everything after it, so check them in order and fix the first one that fails.
- Each box is one gate, in the order a crawler meets it.
- The arrows show the dependency: a later gate never helps if an earlier one fails.
- The last box is the goal: a page an engine can retrieve.
Good sign: AI search bots get status 200 on your key pages in server logs.
Warning sign: new pages stay unindexed, or bots see an empty page without JavaScript.
major AI crawlers rendered JavaScript in Vercel’s study (vendor data)
Vercel, Dec 17, 2024GPTBot requests on Vercel’s network in one month
Vercel, Dec 17, 2024of ChatGPT crawler fetches hit 404 pages
Vercel, Dec 17, 2024of the web is served by Cloudflare, which blocks AI crawlers by default on new domains
Cloudflare, Jul 1, 2025Lessons in Become eligible.
- Lesson 2.1→
Let AI crawlers in
AI crawlers must reach and read your pages before you can be cited. Allow the right bots in robots.txt, render content on the server and fix crawl errors. - Lesson 2.2→
Get indexed
Indexing and technical SEO for AI search: sitemaps with lastmod, IndexNow, server rendering, canonicals and noindex checks that decide AI eligibility. - Research 2.3→
Improve page speed
Pages with first contentful paint under 0.4s averaged 6.7 ChatGPT citations, against 2.1 over 1.13s. See our speed fixes and test plan. - Research 2.4→
Rank in AI Overviews
In eight weeks we fixed ten issues, raised indexing from 42 to 192 of 199 pages and grew pages cited by AI from 2 to more than 100.
See this hub in ThriveStack citedby.
ThriveStack citedby audits crawl readiness, reads your server logs for AI bot visits and errors, and lists each fix with an owner and priority.