robots.txt is the file at the root of your site that tells crawlers which paths they may fetch, and AI providers now run several distinct crawlers you can address by name. Most separate them by purpose: a crawler that collects training data, a crawler that indexes pages for AI search, and a fetcher that visits a page because a user asked about it. You can allow or block each one independently, so you can, for example, stay visible in AI search while opting out of model training.
This post covers how robots.txt works, the main AI user agents, example rules for common policies, and what robots.txt cannot do.
How robots.txt works
The Robots Exclusion Protocol is standardized as RFC 9309. A crawler fetches /robots.txt before crawling, finds the group of rules whose User-agent line matches its name, and follows the Allow and Disallow rules in that group. If no group names it, it follows the User-agent: * group. When several rules match a path, the most specific, meaning the longest, wins.
Two details trip people up. First, a crawler follows only one group: the one that matches it best. If you name GPTBot in its own group, the rules under * no longer apply to it, so repeat any rules it still needs. Second, rules are about paths, not pages you mean: Disallow: /blog also blocks /blog-archive.
The main AI crawlers and what they do
Providers publish their crawler names and purposes. The ones most sites need to decide on:
- OpenAI:
GPTBotfor training,OAI-SearchBotfor search features, andChatGPT-Userfor fetches made on behalf of a user. See OpenAI's crawler documentation. - Anthropic:
ClaudeBotfor training,Claude-SearchBotfor search, andClaude-Userfor user-initiated fetches. - Perplexity:
PerplexityBotfor its search index. - Common Crawl:
CCBot, whose open dataset is widely used to train models. - Google:
Google-Extendedis not a separate crawler. It is a token in robots.txt that controls whether content Googlebot crawls may be used for Google's generative AI models. Blocking it does not affect Google Search. See Google's crawler overview. - Apple:
Applebot-Extendedworks the same way for Apple's models, separate from Applebot.
Names and purposes change over time, so check each provider's current documentation before you rely on this list.
| What it does | Examples | |
|---|---|---|
| Training | Collects content that may be used to train models | GPTBot, ClaudeBot, CCBot |
| Search | Indexes pages for AI search results and citations | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| User-initiated | Fetches a page because a user asked about it | ChatGPT-User, Claude-User |
| Usage tokens | Not crawlers: control use of content already crawled | Google-Extended, Applebot-Extended |
Training, search and user fetches are different decisions
Think about each purpose separately:
- Training: do you want your content used to train future models? This affects what models know, not whether they cite you today.
- Search: do you want to appear, with links, in AI search results? Blocking search crawlers is the fastest way to disappear from those answers.
- User-initiated fetches: when a person pastes your URL into an assistant, do you want it to read the page? Blocking this mostly frustrates your own visitors.
Many businesses choose to allow search and user fetches and to decide training on its own merits. There is no universally right answer; it depends on what your content is and how you earn from it.
Example rules
Allow AI search and user fetches, opt out of training:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Allow: /
Disallow: /admin/
Sitemap: https://example.com/sitemap.xmlRFC 9309 allows several User-agent lines to share one group, as above. Allow everything, but keep private areas out for all crawlers:
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /checkout/
Sitemap: https://example.com/sitemap.xmlYou can also state usage preferences, not just access, with Content Signals: a line in robots.txt that says whether content may be used for search, for AI answers or for training. You can see both on this site's own robots.txt.
Common mistakes
- Blocking everything by accident. A leftover
Disallow: /underUser-agent: *from a staging site hides the whole site from every crawler. - Forgetting group rules do not combine. A crawler that has its own group ignores the
*group, so it can crawl paths you meant to keep private. - Blocking CSS and JavaScript. Crawlers that render pages need those files to see the content.
- Using robots.txt to hide a page from search. A blocked page can still appear as a bare link if others link to it. Use a
noindexmeta tag, on a page crawlers are allowed to fetch, for that. - Syntax errors. Unknown directives or typos can make tools reject the file. Test it after every change.
Review it regularly
New crawlers appear and providers rename or split existing ones. Put a reminder in your calendar to review your robots.txt every few months: check each provider's documentation, compare it with your logs, and adjust the groups. Keep a short comment at the top of the file that explains your policy, so whoever edits it next understands the intent.
What robots.txt cannot do
- It is voluntary. Reputable crawlers follow it; bad actors ignore it. It is not access control.
- It does not remove content already collected. Changes apply to future crawling.
- It is not private. Anyone can read it, so do not list secret paths in it.
- User agents can be spoofed. If you need to verify a crawler, check the provider's published IP ranges where available.
For anything that must stay private, use authentication. Use your server logs to confirm crawlers behave as you expect; our post on measuring visibility in AI answers shows how.
Next step
The free agent-readiness scan checks whether your robots.txt names the AI crawlers, declares Content Signals and points to a sitemap. To see how these files fit together, read llms.txt explained and the agent readiness checklist, or ask us about our SEO service.