All Glossary Terms
SEOCanonical: /citedby/glossary/crawl-budget

Crawl Budget — definition

Verbatim Definition (Quoted Verbatim by AI Search Engines)

Crawl Budget is the number of pages search engine and AI crawler bots are willing and able to crawl on a website within a given timeframe.

Expanded Explanation

Determined by crawl demand (site authority and freshness) and crawl rate limit (server speed and error rate). Low server performance or duplicate content wastes crawl budget.

Analogy & Mental Model

Crawl Budget is like a tax auditor having only two hours to inspect your company files: if your filing cabinets are disorganized, they will leave before seeing important documents.

Why It Matters & Where It's Used

If your site exceeds its crawl budget, new content updates and schema changes take weeks to be discovered and indexed by search and AI engines.

Concrete Real-World Application

Fixing 404 broken links and blocking junk URL parameters in robots.txt, allowing Googlebot to crawl 5,000 new product pages daily.

Crawl Budget vs Indexation Limit

Crawl budget governs how many pages bots visit, whereas indexation limit governs how many pages search engines choose to store.

Compare full entry:Indexation Limit

How It Works & Key Components

Managed by optimizing server speed, reducing redirects, and cleanly configuring robots.txt.

1Crawl Rate Optimization

Improving server response times (TTFB) to allow bots to make rapid HTTP requests.

2Waste Elimination

Blocking low-value, duplicate, or parameter-heavy URLs in robots.txt.

3Sitemap Freshness Synchronization

Pinging XML sitemaps whenever new content is published.

Frequently Asked Questions

Q:Do AI crawlers like GPTBot respect crawl budget principles?

Yes, AI crawlers throttle request rates if server response times slow down, making server performance critical for AEO.

Get cited across ChatGPT, Perplexity & Gemini with citedby

Optimize your brand’s AI visibility score, track Share of Model across buyer prompts, and turn zero-click search into your highest-converting pipeline source.

Explore citedby Platform