Should I block AI crawlers from my website?

The short version

• Crawlers do two different jobs: fetching pages for AI search, and collecting content for AI training.
• Blocking the search crawlers can keep you out of AI answers.
• OpenAI, Anthropic and Google let you opt out of training separately.
• Robots.txt is a polite request that reputable bots follow, not a security lock.
• Check your current file first: some plugins and CDNs block AI bots on their own.

How do I decide, and how do I check my settings?

1. Open yoursite.com/robots.txt in a browser. Look for lines like User-agent: GPTBot followed by Disallow: /.
2. Sort the bots by job. OpenAI’s crawler docs say OAI-SearchBot is for ChatGPT search and GPTBot is for training, and that “each setting is independent.” Anthropic’s crawler article lists ClaudeBot for training and Claude-SearchBot and Claude-User for search and user requests. Google’s crawler list describes Google-Extended, which Google says “does not impact a site’s inclusion in Google Search.”
3. Decide on training separately. Allow the search bots if you want customers to find you. Block the training bots only if that matters to you.
4. Check your security plugin, host and CDN. Some add AI blocks automatically.
5. Recheck after changes. OpenAI notes robots.txt changes for search can take about a day to apply.

Why does this choice matter?

Your website exists to bring in work. When someone asks an assistant for a local recommendation, the tool can only cite pages it’s allowed to read. OpenAI says sites opted out of OAI-SearchBot won’t be shown in ChatGPT search answers. Training is different: it doesn’t send customers to you directly, so opting out of it costs little for most local businesses.

When does blocking make sense, and what are the traps?

Blocking may be right if:

• You sell original work, like courses, articles or photography.
• You have members-only or paid content. That belongs behind a login anyway.
• Bots are overloading a small server. Rate limiting may work better than a full block.

Common traps:

• Blocking everything with one wildcard rule, which can shut out Googlebot too.
• Treating robots.txt as privacy. Google’s robots.txt guide says it’s “not a mechanism for keeping a web page out of Google.” Noindex keeps a public page out of results, but anything private needs a password or login.
• Confusing it with llms.txt, which suggests pages to read but doesn’t control access. See what llms.txt is.

What if my settings aren’t working?

If a blocked bot keeps visiting, check that the user-agent name is spelled exactly as the company publishes it and that the file sits at the root of your domain. If your CDN or firewall blocks bots before they reach robots.txt, change it there. Bots that ignore robots.txt need a firewall rule, not a text file.

How much does it cost, and how long does it take?

Editing robots.txt is free and takes minutes. The effect isn’t instant: each crawler rereads the file on its own schedule.

How does E-3 handle this?

On a site E-3 looks after, tell us which choice you’ve made. People at E-3 make the change, and a robots.txt setting can be switched back later. For the wider work on AI visibility, see our SEO and AI search service.

Related questions

• How do I show up in ChatGPT and other AI answers?
• What are Google AI Overviews, and how do they affect my site?
• What is an llms.txt file, and do I need one?
• What a robots.txt file is (glossary)