A new part of the internet decides what people buy now. You ask ChatGPT for the best tool, the best supplement, the best law firm in your city, and it answers. One name, sometimes three. No list of ten blue links.
Most brands haven’t caught on. Even fewer have checked the one thing that decides whether they can play at all.
Can the AI even read your site?
Not “is your content good” or “is your schema clean”. But something much more boring. When GPTBot or ClaudeBot or PerplexityBot knocks on your server, does it get in, or does it get a wall?
Before any link building or schema markup or any thing else remotely SEO-y, check first whether the engines can reach your pages at all. And one tactic gets this right faster than anything you’ll ever run – the bot-access audit.
Here’s how to run it on your own site.
Every other tactic in AI search assumes that the AI bots can fetch your page. Write a fully-optimised answer-first paragraph, add some Organization schema, maybe even earn a press mention for good measure. But if GPTBot gets a 403, none of it exists.
The block usually isn’t something you even turned on, it’s a default setting.
On July 1st 2025, Cloudflare flipped a switch. Every new domain it onboards now blocks known AI crawlers by default. Cloudflare sits in front of roughly a fifth of the web, so a large slice of the internet moved from “AI can read this” to “AI needs permission” overnight, and most site owners never made that choice.
The reason the block was officially the notion that AI crawlers take content and send almost nothing back. Cloudflare’s own numbers put Anthropic’s ClaudeBot at around 71,000 requests for every referral click it returns. Publishers got tired of being a free training set, so the industry default swung to “no”.
Which is a reasonable default for a news site defending its archive from being ingested for free. But it’s a catastrophe for a brand whose entire goal is to be read, quoted, and named by those same engines.
Requests for your page passes through layers. The network part, your CDN, is the outermost layer. robots.txt is read further in, by the bot, after it has already reached your server. Cloudflare’s block fires at the outermost layer. The crawler is turned away before it ever gets close enough to read robots.txt.
So you can write a perfect robots.txt that welcomes every AI bot on earth, and it changes nothing, because the crawler never reaches the file to read your welcome. The request dies upstream. A brand can have strong content, clean markup, real third-party coverage, a robots.txt that rolls out the red carpet, and still score 0% visibility across every engine.
So here’s the audit. Four steps, thirty minutes, if that, in order.
Step 1: Check Your Bot Settings At The Edge, Not Just In robots.txt
Open your CDN dashboard first. On Cloudflare that’s Security, then Bots (newer plans might show it as AI Crawl Control). You’re looking for a “block AI bots” toggle or a set of Search / Agent / Training categories. On a new zone, assume the block is already on until you’ve seen otherwise with your own eyes. Don’t trust that “I never turned that on” means it’s off. The whole point is that you didn’t turn it on, Cloudflare did.
Step 2: Explicitly Allow The Bots That Cite You
You want two kinds of bot through. The ones that index for retrieval, and the ones that fetch live during a conversation. Allow GPTBot and OAI-SearchBot for OpenAI. Allow ClaudeBot and Claude-SearchBot for Anthropic. Allow PerplexityBot. Allow Google-Extended. And don’t forget the plain search crawlers underneath them, Bingbot and Googlebot, because their index is what several engines read from in the first place. Allow the specific named agents. Leave unknown, unverified scrapers blocked. This isn’t all-or-nothing.
Step 3: Make robots.txt Agree With The Edge
Once the edge lets bots through, robots.txt becomes the layer that gets read. Make sure it doesn’t contradict what you just did. Sometimes, the WAF says “allow GPTBot”, robots.txt says “Disallow: GPTBot”, and you’ve just re-blocked yourself. The two layers have to tell the same story. Edge allows, robots allows, or you’re back to zero.
Step 4: Verify From Outside Your Own Network
This is the step people skip, and it’s the only one that proves anything. Open a terminal and pretend to be the bot:
curl -A "GPTBot" https://yourdomain.com
You want a 200. If you get a 403, the bot is still blocked, and every hour you spend on content is an hour spent on pages the AI can’t open. Run it for each agent that matters: swap GPTBot for ClaudeBot, PerplexityBot, Bingbot. A 403 fails the whole thing so you need to be sure here. And run it against a few different pages, not just the homepage, because a rule can be scoped to a path, so your blog might be reachable while your product pages are walled, or the reverse.
The change isn’t gradual and it isn’t subtle. Before the fix, the engine’s crawler can’t reach the page, so the page can’t be considered, cited, or named. After the fix, the page enters the pool the engine actually draws from. You haven’t guaranteed a citation, but you have made one possible.
Everything downstream builds from here. Every future tactic, the entity-home rewrite, the press mention, the comparison page, only works because the crawler can now fetch what you build. This one check is the foundation the other ninety-nine AI SEO tactics stand on.
Running the curl test gives you a repeatable health check you can trust more than any dashboard. Dashboards tell you what a setting is meant to do, but the curl test tells you what happens when a bot actually appears. Re-run it monthly. CDN defaults change, plans change, a colleague toggles something in a security sprint, a platform ships a new “block AI” default under you. The category defaults are still moving, too; the industry is drifting toward allowing Search-type bots that send referral traffic while blocking training-type crawlers, and those categories get redrawn periodically. The audit that took thirty minutes the first time takes three the second, and three minutes a month is a cheap insurance premium against potentially going dark in AI Search.
