Indexation and Crawl Waste Cleanup for Large Catalogues

On a fashion catalogue of eight thousand products I usually find that most of the URLs Google knows about were never meant to exist. Colour plus size plus sort combinations, session-tagged links, paginated views of filtered views, and two spellings of the same Hebrew category, one percent-encoded and one transliterated. Crawl budget is not the first problem there. Duplication is, and crawl budget is what it costs you. Where the URLs come from: I take ninety days of the Search Console bulk export into BigQuery and join it against your XML sitemaps, your internal link graph and, where they exist, your edge logs. That join answers what the Page indexing report cannot: for every URL Google has seen, is it linked from the site, is it in a sitemap, has it ever received an organic entry, and what status does it return today. Four columns, and nearly every decision follows from them. The four buckets: URLs that should return 410 and leave the link graph. URLs that stay, get canonicalised, and come out of the sitemaps. URLs that should never have been generated at all, which is a template change and not a directive. And the residue that actually matters: pages you want indexed sitting in Discovered, currently not indexed, which is almost never a crawl budget problem and almost always a judgement Google made about value, or a link graph that never reaches them. The order I work in: Stop the generation first. A robots.txt rule on a filter path the template still links to leaves you with a URL Google can index and cannot see, which is worse than what you started with. So facets lose their hrefs before anything else. Then the existing inventory is cleaned. Then sitemaps get segmented by template, so the indexation rate of each is legible on its own line. Then we watch coverage move over six to ten weeks, because the report lags the fix and somebody has to read it correctly. Out-of-stock product pages in fashion and grocery deserve their own decision. Returning 200 with an empty template is the platform default and it manufactures soft 404s at the rate your catalogue turns over. The answer is per product line, not global: seasonal ranges that return, sizes gone for three weeks, and lines discontinued for good are three different answers. Deliverable: A URL decision table with the rule that produced each row, template changes written against your codebase, an edge ruleset for the removals, and a sitemap structure your build can generate. I stay through the first two coverage cycles. Out of scope: Writing or improving the pages that should be indexed. Where a category page is ignored because it holds nine products and forty words, I will name that and stop rather than sell you a fix I do not do. New page types. Internationalisation beyond the Hebrew and English pairs a single Israeli catalogue normally runs. Not for: Sites of a few thousand URLs, where none of this is your constraint and you would be buying process you do not need. Teams who cannot deploy a template change inside a quarter. Anyone who wants the not-indexed count driven to zero, which is not a state any healthy catalogue reaches. What I will not claim: That removing waste raises rankings by itself. It shortens the gap between publishing and indexing, makes coverage readable, and stops you shipping into fog. Where a site had genuinely starved its important templates, indexing of new products moved from days to hours. That is the size of the win and I would rather you buy it knowing that.