Log-driven indexation work, from diagnosis to shipped fix

Between 38 and 61 percent of Googlebot's daily requests on the last six sites we took on went to URLs nobody ever intended to publish: parameter permutations thrown off by faceted navigation, expired inventory held at 200 for internal reasons, session artefacts, pagination tails past page 40 that no human has reached. Each of those requests is one not spent on a page that earns money. This engagement reads your access logs against your index, decides what the request budget should be spent on instead, and stays with the work until the changes are live and the log confirms the shift. Why logs, and not another crawl: A crawl tells you what a bot could reach. A log tells you what Googlebot did reach, at what hour, with which response code, from which of its user agents, and how all of that changed after your last deploy. The two disagree constantly, and the disagreement is where the money sits. Search Console's crawl statistics aggregate the same events into categories too coarse to act on. We work from raw access logs, ideally ninety days, joined to your sitemap set and to URL-level Search Console data, and read the result by template rather than by URL. The first month: Getting the logs takes longer than reading them. Assume two weeks for your infrastructure team to produce a usable export. From the day it lands, ten working days to a decision document: which URL classes stop being crawlable, which stop being indexable, which stay and have their internal linking rebuilt, and which are consolidated. Every line carries the share of requests it currently consumes and the revenue attached to it. Those two figures settle arguments that opinion cannot. What shipping means here: We write the robots directives, the noindex and canonical logic, and the redirect maps ourselves, in your syntax, as pull requests against your repository where your team allows outside commits, or as specifications tight enough that nobody has to interpret them. We review the merge. We sit in the deploy window for anything touching robots.txt or a redirect table, because those are the two changes that can remove a large site from the index before lunch. For the fortnight after release we read the log daily and tell you inside 48 hours if the response distribution moves the wrong way. How we know it worked: Against a measure agreed at kickoff, before anything ships. Usually the share of Googlebot requests landing on the URL set you named as commercially relevant, compared like for like on a weekday-normalised basis. Secondary measures: time from publication to first crawl on new URLs, and the population of the Discovered, currently not indexed bucket in Search Console. We report against those figures whether or not they moved. Twice we have reported a flat result and explained it, both times because the binding constraint was server response time under bot load rather than anything our rules could reach. Outside this scope: Content production, on-page work, link acquisition, international expansion. We do not tune your CDN, though we will tell you when it is the problem, and on two engagements it was. We do not take over your Search Console or run your reporting. We will not ship a change we think is wrong in order to close a ticket, and you will hear that in the thread rather than afterwards. The uncomfortable part: On a large marketplace or retail site the correct answer is often to remove an entire class of pages from the index. Somebody built those pages, defended them in a planning meeting, and still has a number attached to them in a deck. We bring the log evidence into that conversation and will present it to whoever owns the decision, up to and including a board. If your organisation cannot make a decision of that kind, this is the wrong engagement and we would rather say so on the first call than in month four. Not for you if: You cannot obtain raw access logs. Without them this becomes guesswork with extra steps, and we decline the work rather than substitute crawl data and pretend. You want a monthly report. This is engineering work with a search rationale attached, not a reporting subscription. You need a result before your next quarterly review, because the log needs six to eight weeks after a deploy to make a change legible, and no honest reading of it is faster than that.