Crawl Budget — definition
Crawl Budget is the number of pages search engine and AI crawler bots are willing and able to crawl on a website within a given timeframe.
Expanded Explanation
Determined by crawl demand (site authority and freshness) and crawl rate limit (server speed and error rate). Low server performance or duplicate content wastes crawl budget.
Analogy & Mental Model
Crawl Budget is like a tax auditor having only two hours to inspect your company files: if your filing cabinets are disorganized, they will leave before seeing important documents.
Why It Matters & Where It's Used
If your site exceeds its crawl budget, new content updates and schema changes take weeks to be discovered and indexed by search and AI engines.
Concrete Real-World Application
Fixing 404 broken links and blocking junk URL parameters in robots.txt, allowing Googlebot to crawl 5,000 new product pages daily.
Crawl Budget vs Indexation Limit
Crawl budget governs how many pages bots visit, whereas indexation limit governs how many pages search engines choose to store.
How It Works & Key Components
Managed by optimizing server speed, reducing redirects, and cleanly configuring robots.txt.
1Crawl Rate Optimization
Improving server response times (TTFB) to allow bots to make rapid HTTP requests.
2Waste Elimination
Blocking low-value, duplicate, or parameter-heavy URLs in robots.txt.
3Sitemap Freshness Synchronization
Pinging XML sitemaps whenever new content is published.
Frequently Asked Questions
Q:Do AI crawlers like GPTBot respect crawl budget principles?
Yes, AI crawlers throttle request rates if server response times slow down, making server performance critical for AEO.
Get cited across ChatGPT, Perplexity & Gemini with citedby
Optimize your brand’s AI visibility score, track Share of Model across buyer prompts, and turn zero-click search into your highest-converting pipeline source.
Explore citedby Platform