Glossary · Technical SEO & AI Search

Crawl Budget

Definition

Crawl budget is the number of URLs a search engine is able and willing to crawl on your site within a given time, shaped by your server's capacity and how valuable your pages appear.

Google describes crawl budget as the combination of crawl capacity (how much crawling your server can handle without slowing down) and crawl demand (how much Google wants to crawl your URLs based on popularity and freshness). For small stores with a few hundred products, it is rarely a limiting factor. It becomes a real issue for large catalogs, sites with many variants, or stores whose faceted navigation creates huge numbers of URLs.

Filters for size, color, price and sort order can turn a few thousand products into a near-endless set of parameter URLs. If crawlers spend their time there, new and updated pages get discovered more slowly. AI search crawlers face the same problem.

Why it matters for eCommerce

Products that are not crawled cannot rank or appear in AI answers. Wasted crawling delays indexing of new launches, price changes and stock updates, which costs visibility during the moments when demand is highest.

How it works in practice

  • Review crawl stats in Google Search Console and, for large sites, server logs.
  • Keep low-value filter and sort combinations out of the crawl with robots.txt rules, canonicals and clean internal linking.
  • Submit accurate XML sitemaps that list only indexable, canonical URLs.
  • Fix redirect chains, broken links and slow server responses.

Common mistakes

  • Blocking pages in robots.txt that you also want deindexed, so crawlers never see the noindex tag.
  • Leaving out-of-stock or discontinued URLs live as soft 404s.
  • Worrying about crawl budget on a small site instead of fixing content and links.

Put the numbers to work

Our free calculators turn these definitions into your numbers, and the Free Profit Audit shows which ones cost you the most.

Free Profit Audit