Indexation and Crawl Budget Fixes for Faceted Catalogs

On a catalog with faceted navigation the crawlable URL space usually runs ten to forty times the product count, and in the access logs Googlebot typically spends between 55 and 80 percent of its requests inside that space. Almost none of those URLs earn an impression. This engagement redirects that spend, and does it without cutting URLs that quietly bring in revenue, which is the part most cleanups get wrong. What I read, in order: Thirty to ninety days of raw access logs, filtered to verified Googlebot by reverse DNS rather than by user agent string, then grouped by URL template instead of by URL. That grouping is the whole diagnostic: it turns four million lines into fifteen rows, and one of those rows is where your crawl budget went. Then Crawl Stats for host status and response-code mix over time, since a rising share of 5xx or a slowing average response is a throttle on everything else. Then the Page Indexing report for categories, and the URL Inspection API for truth, sampled a few hundred URLs per template, because a category count tells you how many and never which. Sitemaps used as instruments: I split XML sitemaps by template rather than by alphabet or arbitrary chunks. Once each template has its own sitemap, the Search Console coverage numbers become readable per template, and you can watch a single change to the product-detail template propagate without guessing. This is a diagnostic technique first and a submission mechanism second. The decision I take a position on: Disallow in robots.txt stops the request but not the indexing, and it blinds Google to whatever that page links to and to any noindex directive sitting on it. Noindex requires the crawl to happen at all. Canonical is a hint and on parameter URLs a frequently ignored one. Choosing among the three is a judgement about whether you need the crawl spend back or you need the URL out of the results, and those are different problems that get conflated in almost every plan I am asked to review. I will tell you which one you actually have and write the rule accordingly. Expiring inventory: On dealer groups and any catalog with finite stock, the recurring failure is redirecting every sold or delisted item to the homepage or to a generic category. I use a bounded pattern instead: consolidate to the closest surviving page while demand persists, then return 410 once it does not, and keep the model or category page as the durable target. The engagement includes the rule, the window, and the argument for both. Removal is modelled before it ships: Before anything is set to noindex or blocked, I pull sixteen months of Search Console data for the exact URL set proposed for removal. If that set carries a meaningful share of clicks, the plan gets phased and the phases get thresholds. I have ended engagements at this step because the proposed cleanup would have cost more than the crawl efficiency was worth, and saying that in week two is cheaper for both of us than proving it in month six. Not included: Core Web Vitals, JavaScript rendering diagnosis, and content pruning, all of which are separate pieces of work resting on different evidence. No writing of category copy. No implementation, though I will review your pull request against the rule I specified. Who this is not for: Sites under about ten thousand URLs, where crawl budget is not a real constraint no matter what a tool told you. Teams unwilling to change the faceted navigation itself, since parameter rules applied downstream of a nav that generates infinite combinations are maintenance forever. And anyone who wants index bloat cleanup because it sounds like hygiene rather than because the logs show a cost.