How to set robots.txt rules for each AI crawler

The short version

• Robots.txt lives at the root of your domain. One file per host.
• Each bot follows the group that names it most specifically, and ignores the rest.
• Search bots decide whether you can appear in answers. Training controls decide whether your content trains models; Google-Extended also covers Gemini grounding.
• Bots that fetch a page because a person asked may not follow robots.txt at all.
• Test the file after every change. One stray line can block Google.

If you haven’t decided yet whether to block anything, start with our answer on whether to block AI crawlers. This guide is the how-to once you’ve decided.

How do I write the rules?

Diagram: an example robots.txt that keeps AI search bots allowed and opts out of three AI training controls (Google-Extended also covers Gemini grounding). Example values are marked. Checked against Google, OpenAI, Perplexity and Anthropic documentation, October 31, 2026.

1. Open your current file. Visit yourdomain.com/robots.txt in a private browser window. Copy what’s there before you change anything, so you can undo.
2. Find how it’s made. On WordPress, an SEO plugin often generates the file. On a hosted platform, there may be a settings screen. If there’s a real file on the server, you or your developer edit it directly.
3. Keep the default group sensible. The User-agent: * group applies to every bot without a group of its own. Google’s robots.txt guide says rules are case-sensitive and paths start with /.
4. Add a group for each bot you want to treat differently. Start each group with one or more User-agent: lines, then Allow: or Disallow: rules. Use the names exactly as published (see the table below). A bot with its own group ignores the User-agent: * group completely. So repeat any default rule you still want it to follow, such as Disallow: /wp-admin/. In the example diagram, the AI search group would need that line too.
5. Add your sitemap with a Sitemap: line and the full address.
6. Save, then check it loads at the same address in a private window.
7. Check it in Search Console. The robots.txt report shows whether Google fetched and parsed the live file. See what Search Console is for.

The names to use, and what each does

Sources: Google, OpenAI, Perplexity, Anthropic.

Google-Extended has no user agent of its own. Google’s page says crawling uses Google’s existing crawlers, and the token only acts as a control. You’ll never see “Google-Extended” in your server logs, and that’s normal.

Why do the details matter so much?

Robots.txt is short, so one wrong line has a big effect. A Disallow: / in the default group tells every bot without its own group to stay out, including Googlebot and bingbot. On the other side, a typo in a bot’s name means that bot ignores your group and follows the default instead.

Bing has its own controls for AI answers that work through page tags rather than robots.txt. Bing’s 2023 announcement introduced NOCACHE and NOARCHIVE meta tags for its chat answers, and said pages using them still appear in Bing search results. Most local businesses want the opposite of blocking, so they leave these off.

What are the common mistakes and edge cases?

• Expecting robots.txt to hide a page. Google’s robots.txt introduction says it isn’t a way to keep a page out of Google. Use noindex for a public page you just don’t want in results, and a password for anything private. Noindex doesn’t stop anyone with the link opening the page.
• Blocking user-triggered fetchers. OpenAI says “robots.txt rules may not apply” to ChatGPT-User, and Perplexity says Perplexity-User “generally ignores robots.txt rules.” Blocking them in robots.txt may do nothing.
• Forgetting subdomains. A shop at shop.yourdomain.com needs its own file. Google says rules apply only to the exact host and protocol.
• Copying a giant “block all AI” list from a blog. Many of those lists include search bots. Check each name against its company’s page.
• Editing the file but not the firewall. Your host or CDN may block bots before they ever read robots.txt. See security settings that block AI tools.

What if my rules don’t seem to work?

• The bot still visits. Check the spelling and that the file is at the root. Then allow time: OpenAI says about 24 hours for its search systems, and Perplexity says up to 24 hours.
• Google shows an error. Open the robots.txt report in Search Console. It flags lines Google couldn’t parse.
• A bot ignores the file. Robots.txt is a request. Reputable crawlers follow it; others don’t. That needs a firewall rule from your host or CDN.
• You want out of Google’s AI features but not Google Search. Robots.txt can’t do that. Use the setting in our guide to Google’s AI controls.

How much does it cost, and how long does it take?

Free. Writing the rules takes a few minutes; testing takes a few more. Each crawler rereads the file on its own schedule, so expect a day or more before behavior changes.

How does E-3 handle this?

Robots.txt changes are small but risky. We can draft the edit, check every line against the companies’ published names, and keep a copy of the old file so it can be put back. You decide whether it goes live. This sits within our SEO and AI search service.

Related questions

• Where do AI assistants get facts about my business?
• Is my host or CDN blocking AI tools?
• Why set up Bing Webmaster Tools?
• What a robots.txt file is (glossary)
• Back to the complete AI visibility guide

Sources

1. How to write and submit a robots.txt file (Google Search Central)
2. Introduction to robots.txt (Google Search Central)
3. Google's common crawlers (Google Search Central)
4. Overview of OpenAI crawlers (OpenAI)
5. Perplexity crawlers (Perplexity docs)
6. Does Anthropic crawl data from the web? (Claude Help Center)
7. Announcing new options for webmasters to control usage of their content in Bing Chat (Bing Webmaster Blog, September 2023)
8. Bing Webmaster Guidelines (Microsoft Bing)