AI Crawler Access Setup and an Honest llms.txt

We read 90 days of edge logs across seven client origins and found no requests for /llms.txt from GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot or Google-Extended. The file takes an hour and we will ship it if you want it. The work that changes anything is access, and on four of those seven origins the CDN was returning 403 to at least one retrieval fetcher while the marketing team was funding content for the assistants that could not read it. The distinction the toggles hide: Training crawlers and retrieval fetchers are different agents doing different jobs. GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended and Bytespider collect for model training. OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot, PerplexityBot and Perplexity-User fetch a page because somebody asked a question just now and the answer may cite you. You can refuse the first group and welcome the second. Most sites we open have done the reverse by accident, usually through one checkbox in Cloudflare that a security engineer ticked in good faith, sometimes through a WAF rate limit that only bites when a fetcher requests six pages in a second. Google-Extended, specifically: It does not control AI Overviews. Overviews are served from the Googlebot index, and the only levers are nosnippet, data-nosnippet and max-snippet, which cut your ordinary search snippets at the same time. If someone has told you to disallow Google-Extended to keep your content out of AI, they have removed you from Gemini grounding and left Overviews untouched. We reverse that instruction more often than we write a new robots.txt. We verify rather than assume: Reading your robots.txt and declaring it correct is not the job. We request your pages using the actual user agent strings from the published address ranges, record status code and rendered text length, then confirm the same requests in your logs so we know the edge and the origin agree. Client-side rendering surfaces here too: fetchers largely do not execute JavaScript, and a Japanese product page that hydrates in the browser returns a loading shell to the agent that was about to cite it. That finding lands on your front-end team, and we write it in a form they will accept. What ships: A robots.txt with a written reason beside every token, so the next engineer does not undo it. The specific CDN or WAF rule changes, named by rule, for Cloudflare, Akamai or Fastly. A fetch verification matrix covering every agent that matters, before and after. A log query you keep and run yourself each quarter. And llms.txt, if you want it, with our note on what it currently does, which is close to nothing, and the conditions under which that would change. What we will not do: Sell llms.txt as a visibility product. Sell you a blanket block on AI crawlers as brand protection and then sell you visibility work afterwards. Recommend blocking Bytespider or CCBot as though it were a strategy; it is a preference, it costs you nothing and gains you nothing, and we will implement it without pretending otherwise. Who this is not for: Teams who cannot get a CDN change approved. The audit will be correct and nothing will happen, and we would rather you spend the money when the change can actually land. Sites behind an enterprise bot-management contract where the vendor controls the rules and will not expose them; bring the vendor into the room first. And anyone hoping this is the piece that fixes their AI visibility. It removes an obstacle. If the obstacle turns out not to be there, we will tell you in the first week and refund the remainder.