AI crawler access repair, and llms.txt where it earns it

Access is not a content problem, but it is what is broken first. The opening deliverable here is a matrix: every AI fetcher that matters, against every path template you publish, showing the status code your edge actually returns rather than what robots.txt says it should. About one site in five comes back clean. The rest are serving a 403, a JavaScript challenge, an age gate or a geo splash to at least one of GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot, and have been for months, because none of it appears in analytics and nobody gets an alert for a bot that quietly stops coming. Where the blocking actually lives: Rarely in robots.txt. It lives in Cloudflare managed bot rules, AWS WAF Bot Control, Akamai Bot Manager, Fastly, or a Vercel firewall rule somebody added during an incident and nobody removed. We read the rule set itself rather than the dashboard summary, then request each path template with each user agent, from an address inside the region your buyers sit in and one outside it, and keep the raw responses. Three distinctions decide most of the fix: GPTBot, OAI-SearchBot and ChatGPT-User are three fetchers doing three jobs. A legal decision to stay out of training corpora costs you the first; it does not have to cost you the other two, and most sites that made that decision blocked all three. Google-Extended governs Gemini grounding and has never governed AI Overviews, which reads what Googlebot reads, so teams who set it to stay out of AI answers achieved the reverse of their intent. And a page whose body arrives after hydration is retrievable by some fetchers and blank to others; we test that per template rather than reasoning about it. What breaks in your three markets: In igaming it is nearly always the interstitial. Age confirmation and jurisdiction routing return 200 with a page carrying no product content, so what gets stored is the interstitial. Licensed Brazilian operators add redirect hops on the way to the .bet.br host, and every hop is somewhere to lose a fetcher that does not follow the chain. In crypto it is usually a challenge mode switched on during an incident and never switched back. In developer tools it is version-switched documentation behind a client-side route, where the canonical version renders and every other one does not. On llms.txt, we will disappoint you: No assistant we can observe fetches it at answer time, and we will not sell it as a visibility lever. We write one where an agent-facing surface genuinely benefits from a curated index, meaning an API reference, a changelog, a schema catalogue, and where clean markdown of those documents already exists at a stable URL. It points at documents you already maintain. It never becomes a second corpus to keep in sync. If your documentation platform already emits llms.txt and llms-full.txt, most of this work is deleting what it duplicated and repairing the paths inside it. Not included: No rewriting. A page that gets fetched and is still unquotable is a different engagement. No schema work, no link acquisition, no keyword research. And no blanket opening of the site: if your archive is the product, whether to let models read it is a licensing decision that belongs to you, and we will say so and stop rather than make it for you. Who this is not for: Anyone who wants an llms.txt file as the deliverable. Anyone whose valuable pages sit behind authentication, where the honest answer is that this work does nothing for you. How it closes: Edge changes go through your change process. We write the diff and sit in the review; your team merges. Then we wait, because the only evidence that survives a change of agency is thirty days of your own server logs showing the named agents pulling 200s on the templates we opened.