Transcript
Intro
Google wrote a whole guide on crawl budget, and the first thing it does is tell most readers to leave. If your site doesn't have a large number of pages that change rapidly, or your new pages get crawled the same day they're published, you don't need to read this guide. That's Google. So why does every audit tool on earth flag crawl budget?
Because two different problems hide under one name. Crawl budget is how much Google can and wants to crawl, and it only limits big or fast moving sites. Crawl waste is how much of that crawling goes to URLs that should never have existed, and that one hits sites of any size. Today we pull them apart, then go after the biggest source of waste: filters, sorting and pagination.
Who needs to care
So who is the guide actually for? Google names three kinds of site. More than a million unique pages changing about weekly. More than ten thousand pages changing daily. Or a site where a large share of URLs sit in Search Console as discovered, currently not indexed. And Google calls even those rough estimates, not thresholds.
How do you know which side you're on? Open the page indexing report and compare the discovered, currently not indexed count with your total. A few stragglers are normal. A big and growing share means Google knows about URLs it hasn't got round to fetching. Then check crawl stats. If new pages get fetched within a day or two, budget isn't your limit.
Which makes our first myth easy. Every site has a crawl budget problem, so every site needs crawl budget work.
Google's own scope says otherwise, and it adds something people forget. Back in 2017, Google wrote that crawling is necessary for being in the results, but it's not a ranking signal. A higher crawl rate doesn't lift a page. For a clinic with a few hundred pages, or a local restaurant, crawl budget is almost never the bottleneck. Verdict: placebo.
Budget vs waste
Here is the model Google publishes. Two things set your budget. Capacity is how much Google can fetch without hurting your server. Demand is how much it wants to fetch, driven by how popular your URLs are and how stale its copies get. If demand is low, Google crawls less, even when capacity is sitting there unused.
Notice the phrase URL inventory on the demand side. Demand depends partly on how many URLs Google thinks you have, and that is where waste comes in. In 2017, Google ranked the low value URLs that hurt crawling, most significant first. Top of the list: faceted navigation and session identifiers. Then duplicate content, soft error pages, hacked pages, infinite spaces and spam.
Session identifiers share that top spot, and they still turn up. If your site adds a session or tracking parameter to its own links, every visit can mint a brand new URL for the same page. Your site is quietly printing duplicates. Keep sessions in cookies, keep tracking tags off internal links, and the inventory stops growing by itself.
And that is the problem small sites do have. A site with nine thousand real pages can expose millions of URLs if every filter makes a new one. Google has to discover them, work out what they are and keep coming back. Meanwhile your new pages wait in the same queue, behind two million variations of thrillers sorted by runtime.
Parameters cost even when the content is identical. In a 2024 study of Googlebot on one large documentation site, URLs with query strings waited longer to be rendered. The median was 13 seconds against 10. At the slow end, about eight and a half hours against two and a half. Tracking tags on your own internal links create exactly these URLs.
Server health sets the pace
The capacity side is set by your server, more than by anything in Search Console. Google says the limit rises when your site answers quickly and consistently, and falls when it slows down, throws server errors, or returns 429, too many requests. Each hostname has its own budget, so a subdomain is counted separately from your main site.
Crawl stats shows this directly. Look at the average response time trend, and at host status, which flags trouble fetching robots.txt, resolving DNS or connecting to your server. If response time climbs every evening while your backups run, Google sees a slower site every evening too. It doesn't know about your backups. It just backs off.
There used to be a setting for this. Search Console's crawl rate limiter was switched off on January 8, 2024. Google said that when a server keeps returning 500 errors, Googlebot slows down automatically and almost immediately. Handy in an emergency. The flip side is less handy: a flaky server quietly throttles your own crawling every single day.
Two more cheap server checks. Google supports 304 not modified, which lets it reuse its copy of an unchanged page instead of downloading it again. And its Googlebot documentation, updated in February 2026, says it crawls the first two megabytes of each HTML file. A page stuffed with inline scripts and data can push real content past that cut off.
Redirect chains are the quiet version of the same waste. Google says long chains have a negative effect on crawling. The usual cause is internal links still pointing at old URLs after a redesign, so every click and every crawl takes two or three hops. Point the links at the final address, and the hops disappear.
Finding them is quick. Crawl the site, export every internal link that lands on a redirect, and group the list by the template the link sits in. It's rarely a thousand separate problems. Usually it's one footer, one menu or one related links block, still pointing at last year's URLs. Fix the template, and the whole list empties at once.
Facets
Now the facets. ReelRadar, our fictional film site, has about nine thousand films and six filters: genre, streaming service, decade, runtime, age rating and sort order. Count every combination, including leaving a filter unset, and those six filters can produce over two million URLs. That's more than two hundred filter pages for every real film.
Crawl stats tells you if this is happening to you. It splits Google's requests by response, file type and purpose, and every row opens into example URLs. On ReelRadar, almost three quarters of Googlebot's requests in a month went to filter URLs. The film pages, the ones that actually earn traffic, got less than a fifth.
Google's faceted navigation guide gives you two roads. If you don't need filtered URLs in search, stop them being crawled at all, with robots.txt rules, or by putting filters in the URL fragment after the hash, which Google generally ignores. Canonical tags and nofollow on filter links exist too, and Google says they're generally less effective in the long term.
But some filter pages are real landing pages. People do search for thrillers on Netflix, or horror films from the eighties. So the call is made per facet, from demand data, not from a rule of thumb. We build a facet decision matrix: every filter and every pairing gets a row, a demand check and one of three outcomes.
Where does the demand check come from? Start in Search Console. Filter the queries for searches that combine a genre with a service or a decade, and see which pairings already earn impressions. Then use a keyword tool for the pairings you don't rank for yet. A pairing nobody searches for anywhere doesn't need an indexable page.
Here is ReelRadar's matrix. Genre pages: indexable, self canonical, in the menu. Streaming service pages: indexable. Genre by service: indexable only for pairs with real demand, like thrillers on a big service, linked from the genre page. Decade, runtime, rating and sort order: kept out of crawling entirely, in any combination.
The blocking itself is a few lines. Google's guide shows the pattern: disallow URLs that carry each filter parameter. On ReelRadar, that's one disallow each for decade, runtime, rating and sort, while genre and service pages live on clean paths that stay crawlable. Five minutes of typing, and two million URLs stop being invited in.
For the facets you keep crawlable, Google adds some housekeeping. Use the standard ampersand between parameters. If filters live in the path, keep them in the same order every time, so thriller on netflix and netflix thriller never both exist. And never allow the same filter twice in one URL.
The last rule deserves its own moment. If a filter combination has no results, Google says return a real 404 status code. Not a friendly no films found page with a 200 status, and not a redirect to a shared not found page. Same for duplicate filters, nonsense combinations and page numbers that don't exist.
On ReelRadar, the westerns on the kids streaming service page has nothing on it, because there are no such films. Shocking, we know. Before the fix it showed a friendly message with a 200 status, so Google kept a soft error page in its inventory and kept coming back. After the fix it returns 404, which Google treats as a strong signal not to crawl that URL again.
Quick quiz, pause if you like. Three ReelRadar URLs: thrillers on Netflix, thrillers sorted by rating, and westerns on the kids service. Index, block or 404?
Thrillers on Netflix gets indexed, because people search for it. The sorted list gets blocked, because nobody searches for a sort order. The westerns page returns 404, because it's empty. Three URLs, three different answers.
Pagination
Next myth. Paginated lists should point their canonical tag at page one, so all the authority collects there.
Google's pagination guidance says the opposite, in plain words. Don't use the first page of a paginated sequence as the canonical page. Give each page its own canonical URL. Page three lists different films from page one, so it isn't a duplicate. Canonicalise it away and you ask Google to ignore the only path to the films on it. Verdict: placebo.
The rest of the pagination rules are short. Every page gets a unique URL, like question mark page equals three. Pages link to each other with real links. No hash fragments for page numbers, because Google ignores them. And Google stopped using rel next and rel prev, though other engines may still read them.
Something you can stop worrying about: unique titles on every paginated page. Google says pages in a sequence can share a title and description, because it tries to recognise the sequence. Sorted copies of the same list are a different story, and Google suggests keeping those out of the index.
Load more buttons and infinite scroll need care. Google's crawlers don't click buttons and generally don't trigger JavaScript that needs a user action. If your list only grows when someone clicks or scrolls, Google may only ever see the first screen. Keep the button for people, and back it with paginated URLs crawlers can follow.
Noindex vs robots.txt
Last myth, and the one we find most often in audits. Noindex the filter pages, and you'll save crawl budget.
Think about how noindex works. It's a tag inside the page, so Google has to fetch the page to read it. Google's crawl budget guide says it directly: don't use noindex, as Google will still request the page and then drop it, wasting crawling time. Noindex controls the index. It does nothing for crawling. Verdict: placebo.
The crawl control is robots.txt, and it has its own catch. Block a URL in robots.txt and Google never sees any noindex on it, so the bare URL can still appear in results if other pages link to it. So pick one tool per job. robots.txt for URLs you never want crawled. Noindex for pages Google may crawl but shouldn't index.
And don't use robots.txt as a dial. Google says blocking some pages won't move budget to others, unless you're already at your capacity limit. Blocked URLs also linger in the queue and get recrawled when the block lifts. For pages that are gone for good, a 404 or 410 is the cleaner exit.
Crawl stats has one more useful split: crawl purpose. Discovery means Google is fetching a URL it has never seen. Refresh means it's rechecking a known one. After a facet fix, you want discovery spent on real new pages, not on yet another filter combination, so open the example URLs behind that row and read them.
Your sitemap is the other half of the inventory you hand Google. Google reads it regularly, so it should list the pages you want crawled, with honest lastmod dates for updated content. On ReelRadar, the sitemap now holds film pages, genre and service hubs, and the indexable pairings from the matrix. No filters, no sorts, no empties. Ever.
Wrap up
So here is the order we work in when crawling looks wrong. Host status and server errors first. Then the share of requests going to filter, sort and parameter URLs. Then soft 404s and redirect chains. Only after all that do we ask whether the site is big enough for crawl budget itself to matter. Usually by then, nobody's asking.
The scorecard. Every site having a crawl budget problem: placebo. Empty filter pages being harmless: placebo. Canonicalising every page to page one: placebo, and it hides your own listings. Noindex saving crawl budget: placebo. What works is the plain stuff: a fast, healthy server, filters that exist only where people search for them, and real 404s for everything empty.
If you do one thing this week, open crawl stats and click into the response codes and file types. Read the example URLs Google shows you. If you see filter parameters, sort orders or empty listings in there, those exact URLs are the first rows of your facet matrix.
The facet decision matrix template is free on the episode page at irongoo.com. List your filters, check the demand for each pairing, and decide index, block or 404 before your next crawl report decides for you.
