Crawl Budget

The web is larger than any crawler can explore, so there are limits to how much time and resources Google can spend on any one site. Google calls the allocation a site’s crawl budget, treats each hostname as a separate site with a budget of its own, and says the budget is set by crawl capacity limit and crawl demand^[1]. It defined the term in 2017 as the number of URLs Googlebot can and wants to crawl, and the post now points to its updated documentation on optimizing crawling efficiency^[2].

Capacity is about the server. The limit goes up when a site responds consistently and fast, and down when it slows, answers with server errors or sends rate-limiting signals such as HTTP 429^[1]. Demand is about the site. For Googlebot it varies with the size of the site, how often it is updated, the quality of its pages and their relevance. The factor a site controls most is its perceived inventory: when many known URLs are duplicates or are unwanted, Google spends crawling time on them^[1].

Most sites do not need to manage it. Google’s guide is for sites with a million or more unique pages that change weekly, sites with ten thousand or more that change daily, and sites where a large share of URLs sit in Search Console as discovered but not indexed, and it says these figures are rough estimates, not thresholds^[1]. For a site where pages are crawled the day they are published, keeping the sitemap up to date and checking the Page indexing report is enough^[1].

Crawling is not ranking. Google said in its 2017 post that crawling is necessary for being in the results but is not a ranking signal, and that a higher crawl rate does not by itself give better positions^[2]. What a limited budget costs is delay: new and changed pages are found and refreshed later than they should be.

Before Google crawls a site it checks for a recent successful robots.txt response, and when the request fails it slows or stops crawling until it gets an acceptable answer^[3]. Whether a request counts as failed depends on who asks and on the status it gets: Google treats 403, 404 and 410 as a successful robots.txt response^[3]. QueryRev’s study used its own crawler, not Googlebot, and 3,679 of the 9,973 domains, 36.9%, gave it no robots.txt answer that it could read, among them names that did not resolve and sites that refused it with an access error^[4]. The figure describes that crawler’s view, not how often Googlebot’s request fails.

Search Console’s Crawl Stats report shows Google’s crawling history for a site: total crawl requests, total download size, average response time, host status, and requests grouped by response, file type, crawl purpose and Googlebot type. It is available for root-level properties and is aimed at advanced users; a site with fewer than a thousand pages should not need it^[3].

Read it with care. The report counts the URLs actually requested, not their canonicals, and counts every hop of a redirect chain as a request of its own^[3]. Google also says some requests may not be counted, so the report can differ slightly from a site’s own server logs^[3]. The logs also show crawlers other than Google’s.

Group the requests by the kind of URL they went to and look at the split. The questions are how many requests went to pages the site wants in search, how many to filter, sort and internal search URLs, and how many ended in redirects or errors. Then measure the delay that matters: the days between a page being published and its first crawl.

Sample data: an invented site, to show the method.

A sample store with about 600,000 URLs, most of them filter and sort combinations, reads one day of its server logs. A search crawler made 10,000 requests. Of these, 5,200 went to filter and sort URLs, 1,100 to internal search pages and 700 ended in redirects or errors. The 2,100 product pages and 900 category pages the store wants crawled received 3,000 requests between them, and a new product takes nine days to be crawled for the first time.

The store acts on the split. It consolidates the filter combinations that duplicate a category, stops linking to the rest, blocks internal search with a robots.txt rule, replaces internal links to redirects with their targets and keeps its sitemap up to date. A month later it reads the same split from the same logs, and judges the work by the days to first crawl.