What Crawl Budget means
Crawl budget combines two things: crawl rate limit (how fast a crawler can fetch without degrading your server) and crawl demand (how much the engine actually wants your content, driven by popularity and freshness).
For most sites under a few thousand URLs it isn't a real constraint. It becomes one on large catalogs, faceted navigation, and sites generating parameter URLs at scale, where crawlers can spend most of their allocation on near-duplicates.
The symptom is indirect: important pages get crawled infrequently, so updates take a long time to register and structured data in the index goes stale.
Why it matters for AI search
AI retrieval raises the stakes. A product page crawled once a quarter carries pricing and availability that may be months out of date, and models will state that stale data confidently.
It also affects the newer AI crawlers, which generally have less patience and lower tolerance than Googlebot. Budget wasted on junk URLs is budget not spent on the pages you want cited.
See it in action
A 6,800-product catalog with uncontrolled faceted navigation. Watch where the crawl goes.
Product pages went from 9% of crawl to 52%.
Nothing was added. Facets were blocked from crawling, parameters canonicalised, and the freed allocation went to pages that earn revenue. Product data in the index started refreshing weekly instead of quarterly.
How to get it right
Reclaiming wasted crawl
- Audit server logs to see where crawlers actually spend their time — not where you assume they do
- Block or canonicalise faceted navigation and parameter URLs that produce near-duplicates
- Retire or consolidate thin and legacy pages rather than letting them accumulate
- Fix redirect chains; every hop costs a fetch that could have gone to a real page
- Keep sitemaps accurate and current so discovery doesn't depend on crawling through the site
Common questions
Does crawl budget matter for small sites?
Rarely. Under a few thousand URLs, crawlers generally fetch everything they want without constraint.
It becomes a genuine issue on large catalogs, sites with faceted navigation, or anywhere URL generation has run away.
Do AI crawlers have their own crawl budget?
Effectively yes, and generally a smaller one. GPTBot, ClaudeBot and PerplexityBot are less established than Googlebot and tend to fetch less deeply.
That makes waste more costly — a crawler with a limited allocation spending it on facet URLs may never reach your product pages at all.
How do I know if I have a crawl budget problem?
Check Search Console's crawl stats against your real page count, and look at how long updates take to appear in the index. Server log analysis gives the clearest picture.
The classic signal is a large gap between submitted and indexed URLs alongside heavy crawling of parameter URLs.
Will blocking pages hurt rankings?
Blocking genuinely low-value pages usually helps, because it concentrates crawl on pages that matter.
The risk is careless blocking — disallowing something that was ranking, or blocking resources needed to render pages. Audit before you block.
Related concepts
These come up alongside Crawl Budget constantly.