Is your host or CDN blocking AI tools without you knowing?
Possibly. Security plugins, hosting firewalls and CDNs can block AI crawlers on their own, and some add AI rules to robots.txt for you. Cloudflare, for example, now has separate policies for AI search, agent and training bots. Check robots.txt, your firewall or bot settings and your server logs, then set each category on purpose.
The short version
- Robots.txt isn’t the only gate. Firewalls and CDNs can turn bots away first.
- Some services add AI rules to your robots.txt automatically.
- Cloudflare now sorts AI bots into Search, Agent and Training, each with its own setting.
- Check three places: the live robots.txt, your security or CDN dashboard, and your server or CDN logs.
- Allow search bots if you want to appear in AI answers; decide training separately.
How do I check what’s being blocked?
1. Read the live robots.txt
Open yourdomain.com/robots.txt in a private window. Look for lines you didn’t write, such as groups for GPTBot, ClaudeBot or Google-Extended, or a “Content-signal” line. Cloudflare’s managed robots.txt page says its Bot Preference Sync can add directives like these based on your bot policies, and its example disallows Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent.
2. Check your CDN or firewall settings
Every service labels this differently, so look for “bots,” “AI,” “crawlers” or “firewall” in its dashboard. For Cloudflare, the AI bots page describes:
- Security Settings > Configure AI bot policies, with separate categories for Search, Agent and Training.
- For each category: Block (on all pages), Block on pages with ads, or Allow (do not block).
- An older Block AI bots setting, which Cloudflare lists as deprecating on September 15, 2026.
Cloudflare also says that from September 15, 2026, new domains have Training and Agent bots blocked on pages with ads, with Search allowed. If your site has no ads, check what your settings actually show rather than assuming.
3. Check your security plugin
WordPress security and firewall plugins may include bot or rate-limit rules. Look for lists of blocked user agents and for “block AI crawlers” toggles. Note what’s on before changing anything.
4. Look at the logs
Your host or CDN may show requests by user agent. Search for the names you care about: OAI-SearchBot, PerplexityBot, Claude-SearchBot, bingbot, Googlebot. A status code of 403 for one of them means something is turning it away. Ask your host if you can’t find the logs.
A name in the logs isn’t proof of who sent the request. Anyone can copy a user agent string. Cloudflare’s verified bots page checks identity with published IP ranges or a cryptographic signature (Web Bot Auth), not the name alone. Before you allow or block by name, check the request against the company’s published IP ranges, or use your CDN’s verified-bot setting.
5. Decide each category on purpose
- Search bots fetch pages so AI tools can cite them. OpenAI says sites opted out of OAI-SearchBot won’t appear in ChatGPT search answers, and Perplexity recommends allowing PerplexityBot to appear in its results.
- Agent bots act for a person, such as when an assistant opens a page because someone asked it to. Blocking them may stop an assistant from reading your page when someone asks about you.
- Training bots collect content for model training. Blocking them doesn’t take you out of search answers, according to the companies that separate the two.
Why does this matter?
Google’s AI features page lists, as its first best practice, allowing crawling “in robots.txt, and by any CDN or hosting infrastructure.” That second part is easy to miss. A firewall that blocks a crawler means your carefully written robots.txt is never read, and that tool can’t fetch fresh copies of your pages. Blocking isn’t a clean removal either: a tool may still use an older copy it already holds, or describe you from other sites that mention you.
What are the common mistakes?
- Checking robots.txt only. The firewall can block first.
- Turning on a “block all bots” switch to deal with spam, and taking out search crawlers with it.
- Blocking by IP from an old list. Companies publish current IP ranges: OpenAI’s are linked from its crawler page, and Perplexity’s from its bot page. Old lists go stale.
- Assuming defaults suit you. A default is a guess about a typical site. A local service business usually wants the search category allowed.
- Changing settings without noting what they were. Write down the old values so you can undo.
What if a bot is overloading my server?
That’s a real problem, especially on small hosting plans. Rate limiting, which slows a bot rather than blocking it, is often a better first step than a full block. Your host or CDN can set it. Bots that ignore robots.txt entirely need a firewall rule; Cloudflare’s own page notes robots.txt compliance is voluntary.
How much does it cost, and how long does it take?
Checking is free and takes about half an hour. Changing a CDN setting takes minutes. Some advanced bot controls are only in paid plans; check your provider’s pricing page.
How does E-3 handle this?
We look at robots.txt, firewall and CDN bot settings together for clients on our SEO and AI search service, and explain each category in plain terms so the client can choose. We can note the old settings before any change so they can be put back. You decide what changes.
Related questions
Next step
- Next: Do you need GEO, special AI markup or AI-only pages?The next guide in AI visibility.
- Free website checkPublic checks on your own site, then an action plan of what to fix first.
Still stuck? Ask E-3 for an answer from our help guides.
Rather we handled ai visibility for you?
It’s part of what we do every week for small businesses, as part of seo and ai search. Start with a free check of your site and profile, or call and ask.