Edge Log Analysis of Googlebot Behaviour on Retail Sites

If your site sits behind Cloudflare, Akamai or Fastly, your origin logs are already a partial record: the edge answered a share of Googlebot's requests and your application never saw them. So the first thing I ask for is edge logs. On the last three retail catalogues I read, between forty and sixty per cent of verified Googlebot requests went to URLs the team did not want indexed. Getting the data: Logpush to R2 or S3, DataStream 2, or Fastly streaming, depending on what you run, then into BigQuery. Thirty days gives a first read of crawl distribution. Twelve months is what you need to see seasonality, the crawl response to a release, or how discovery lag has changed since last year. If your retention is seven days, we start the retention change and I come back, rather than me charging you to over-interpret a week. Verification before anything else: Every hit claiming to be Googlebot goes through reverse DNS and a forward confirmation against googlebot.com and google.com. On consumer retail domains a meaningful share of claimed Googlebot traffic is not Google at all, and every report built on user agent strings alone has been wrong in both directions. What logs answer that no crawler can: Which templates Google actually spends requests on, day by day. Whether new products are discovered in hours or in a fortnight, measured as the gap between publish time and first fetch. Whether conditional requests work, because a catalogue answering 200 to every If-Modified-Since is paying to re-serve pages that have not changed. How the smartphone and desktop crawlers split on your site. Which URLs are fetched repeatedly and linked from nowhere, which is how you find the parameter a filter emitted for a year. And the relationship between origin response time and daily crawl rate, which is the closest thing to a crawl budget control most sites actually have. For logistics and freight sites the recurring finding is different: tracking and quote URLs generated per session, fetched indefinitely, ranking for nothing, consuming the requests that should be going to service and lane pages. What you get: The queries themselves, in your BigQuery project, so the analysis is repeatable without me. A written read of what the crawl distribution says about the site. And the three or four changes that would move it, each written as a ticket with the log query that will show whether it worked. What I will not sell: A monthly log report. If a retainer's product is a document, it is the wrong retainer. Ongoing work here exists to catch crawl regressions after releases and to close the loop on tickets already open, and if a month passes with nothing worth acting on I would rather tell you that than fill a page. Not for: Sites of a few thousand stable URLs, where the finding will be that crawling is fine and you will have paid to hear it. Teams who cannot get log access out of infrastructure inside a month, which is the most common reason this work stalls before it starts. Anyone expecting a ranking narrative: logs tell you what was fetched and say nothing about why a page ranks where it does. One honest limit: Logs are a record of the past. They show where crawl went and never where it should have gone. That judgement is what you are paying for, and I will show the reasoning rather than hand you a number I cannot defend.