robots.txt
A small text file at the root of your website that tells search engine and AI crawlers which parts of the site they may visit.
robots.txt is a plain text file at yoursite.com/robots.txt. It gives instructions to automated crawlers, such as Googlebot or AI crawlers like GPTBot, about which parts of your site they may or may not crawl. Each section names a crawler (a “user-agent”) and lists paths it’s allowed or disallowed.
Google’s robots.txt guide explains the format.
Why it matters to a small business
One wrong line in robots.txt can stop search engines crawling your entire site. A common mistake is launching a new website with the staging site’s robots.txt, which blocks everything.
It also controls AI crawlers. Some security tools and hosts add AI-blocking rules automatically, which may keep you out of AI search answers without you knowing.
Two important limits:
- It’s a request, not a lock. Reputable crawlers follow it; bad bots may ignore it. Don’t use it to hide private information.
- It controls crawling, not indexing. A blocked page can still appear in Google if other sites link to it. To keep a page out of search results, use a noindex tag instead, and don’t block it in robots.txt, or Google can’t see the noindex.
Example
A contractor’s traffic drops after a redesign. Their robots.txt contains “User-agent: *” and “Disallow: /”, left over from the staging site. Removing that line and resubmitting the sitemap in Search Console lets Google crawl the site again.
Learn more
Rather talk it through?
Call our team, or send a note and we’ll reply within one business day. No pressure and no jargon.