Marketplace index architecture for daily-changing supply

On a marketplace where supply turns over daily, the number that decides everything else is the median lifetime of a listing URL. If half your listings disappear within three weeks and each returns a 404 the day it is delisted, most of what search engines crawl is already dead, and the category pages that could rank are recrawled too rarely to move. The fix is not more listings in the sitemap. It is deciding, per listing type, which URLs are permanent, which expire, and what an expired one becomes. Where the crawl budget actually goes: The first week is spent in log files rather than in a crawler. Raw access logs show which URL patterns Googlebot spends its requests on, and on marketplaces the answer is almost always sorted and filtered permutations of category pages that nobody linked to and nobody should have allowed into the index. Search Console's crawl stats report is too aggregated to show pattern-level waste, so we parse the logs directly. We measure the split before proposing anything, because the argument with your engineers will be about whether facet URLs matter, and that argument is easier to win with their own logs. The expiry decision: Every listing type gets one of three rules. Permanent, because the URL describes something that recurs: a model, a route, a role at a company that hires continuously. Expiring to a parent, where the listing returns a 301 to its category or seller page and the category absorbs the demand. Or never indexed at all, which is the right answer for a large share of one-off supply and the answer clients resist hardest. We write the rule, the HTTP status for each state, and what a human arriving from an old link is shown. The category layer: Categories are the durable asset and they are usually built as inventory dumps. We restructure them around the query a buyer actually types, which on most marketplaces is a two- or three-facet combination, and specify which combinations get a static, linked, indexable page and which stay behind a canonical. The rule we work to: a facet combination earns a URL when it holds enough supply to stay non-empty in your leanest month, not your fullest. Empty-state handling is part of the specification, because a category that empties and stays indexed does more damage than one that never existed. Who runs it: Led by the practitioner who handles our page-system work, with a second reviewer on the technical specification. Prior calls include telling a European classifieds operator to deindex roughly half its category tree, which cost a third of its indexed URL count and returned the traffic inside two quarters. We present that kind of recommendation to your leadership ourselves rather than handing you the slide. When the answer is unwelcome: Marketplace teams are measured on live inventory, so a finding that most listings should never enter the index reads as an attack on the metric people are paid against. We will still say it, with the log evidence attached. Where a recommendation is overruled we write down what was overruled and what we expect to happen, so the next review starts from a record instead of a debate. Out of scope: Listing copy and seller-supplied content quality: we specify the required fields and the thresholds, we do not rewrite listings. Internal search relevance and marketplace ranking logic, which is a product problem and not ours. Paid acquisition. Review markup is specified only where it is legitimate for your model, and we will not write structured data that misrepresents who left a review. Not for: Single-vendor catalogues, where the inventory logic is different and cheaper to solve elsewhere. Marketplaces below roughly five thousand live listings, where hand-built category pages beat any system we would design. Teams without server log access or without the ability to change the HTTP status returned on delisting, because without those two this is theory rather than a plan. How the retainer differs from the project: The project ends with a specification. The retainer keeps a hand on it while inventory shifts: monthly log review, index counts checked against the thresholds we set, and the call on new facet combinations as your supply mix moves, because a combination that earned its own URL in March may not hold in November.