# robots.txt for AI crawlers: GPTBot, ClaudeBot and the rest > How to write robots.txt rules for AI crawlers: the main user agents, training versus search versus user-initiated fetches, example rules, and robots.txt limits. Canonical page: https://aispeedforce.com/blog/robots-txt-for-ai-crawlers/ Last updated: 2026-09-23 AI providers run separate crawlers for training, for search and for fetching a page a user asked about. robots.txt lets you treat each one differently. Here is who is who and how to write the rules. [SEO](https://aispeedforce.com/blog/category/seo/) Published 23 September 2026 5 min read By AI SpeedForce robots.txt is the file at the root of your site that tells crawlers which paths they may fetch, and AI providers now run several distinct crawlers you can address by name. Most separate them by purpose: a crawler that collects training data, a crawler that indexes pages for AI search, and a fetcher that visits a page because a user asked about it. You can allow or block each one independently, so you can, for example, stay visible in AI search while opting out of model training. This post covers how robots.txt works, the main AI user agents, example rules for common policies, and what robots.txt cannot do. ## How robots.txt works The Robots Exclusion Protocol is standardized as [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309). A crawler fetches `/robots.txt` before crawling, finds the group of rules whose `User-agent` line matches its name, and follows the `Allow` and `Disallow` rules in that group. If no group names it, it follows the `User-agent: *` group. When several rules match a path, the most specific, meaning the longest, wins. Two details trip people up. First, a crawler follows only one group: the one that matches it best. If you name GPTBot in its own group, the rules under `*` no longer apply to it, so repeat any rules it still needs. Second, rules are about paths, not pages you mean: `Disallow: /blog` also blocks `/blog-archive`. How a crawler applies robots.txt ## The main AI crawlers and what they do Providers publish their crawler names and purposes. The ones most sites need to decide on: - **OpenAI**: `GPTBot` for training, `OAI-SearchBot` for search features, and `ChatGPT-User` for fetches made on behalf of a user. See [OpenAI's crawler documentation](https://platform.openai.com/docs/bots). - **Anthropic**: `ClaudeBot` for training, `Claude-SearchBot` for search, and `Claude-User` for user-initiated fetches. - **Perplexity**: `PerplexityBot` for its search index. - **Common Crawl**: `CCBot`, whose open dataset is widely used to train models. - **Google**: `Google-Extended` is not a separate crawler. It is a token in robots.txt that controls whether content Googlebot crawls may be used for Google's generative AI models. Blocking it does not affect Google Search. See [Google's crawler overview](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers). - **Apple**: `Applebot-Extended` works the same way for Apple's models, separate from Applebot. Names and purposes change over time, so check each provider's current documentation before you rely on this list. | | What it does | Examples | |---|---|---| | Training | Collects content that may be used to train models | GPTBot, ClaudeBot, CCBot | | Search | Indexes pages for AI search results and citations | OAI-SearchBot, Claude-SearchBot, PerplexityBot | | User-initiated | Fetches a page because a user asked about it | ChatGPT-User, Claude-User | | Usage tokens | Not crawlers: control use of content already crawled | Google-Extended, Applebot-Extended | Three kinds of AI fetcher ## Training, search and user fetches are different decisions Think about each purpose separately: - **Training**: do you want your content used to train future models? This affects what models know, not whether they cite you today. - **Search**: do you want to appear, with links, in AI search results? Blocking search crawlers is the fastest way to disappear from those answers. - **User-initiated fetches**: when a person pastes your URL into an assistant, do you want it to read the page? Blocking this mostly frustrates your own visitors. Many businesses choose to allow search and user fetches and to decide training on its own merits. There is no universally right answer; it depends on what your content is and how you earn from it. ## Example rules Allow AI search and user fetches, opt out of training: `User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot User-agent: Google-Extended User-agent: Applebot-Extended Disallow: / User-agent: * Allow: / Disallow: /admin/ Sitemap: https://example.com/sitemap.xml` RFC 9309 allows several `User-agent` lines to share one group, as above. Allow everything, but keep private areas out for all crawlers: `User-agent: * Allow: / Disallow: /admin/ Disallow: /checkout/ Sitemap: https://example.com/sitemap.xml` You can also state usage preferences, not just access, with [Content Signals](https://aispeedforce.com/blog/content-signals-explained/): a line in robots.txt that says whether content may be used for search, for AI answers or for training. You can see both on this site's own [robots.txt](https://aispeedforce.com/robots.txt). ## Common mistakes - **Blocking everything by accident.** A leftover `Disallow: /` under `User-agent: *` from a staging site hides the whole site from every crawler. - **Forgetting group rules do not combine.** A crawler that has its own group ignores the `*` group, so it can crawl paths you meant to keep private. - **Blocking CSS and JavaScript.** Crawlers that render pages need those files to see the content. - **Using robots.txt to hide a page from search.** A blocked page can still appear as a bare link if others link to it. Use a `noindex` meta tag, on a page crawlers are allowed to fetch, for that. - **Syntax errors.** Unknown directives or typos can make tools reject the file. Test it after every change. ## Review it regularly New crawlers appear and providers rename or split existing ones. Put a reminder in your calendar to review your robots.txt every few months: check each provider's documentation, compare it with your logs, and adjust the groups. Keep a short comment at the top of the file that explains your policy, so whoever edits it next understands the intent. ## What robots.txt cannot do - **It is voluntary.** Reputable crawlers follow it; bad actors ignore it. It is not access control. - **It does not remove content already collected.** Changes apply to future crawling. - **It is not private.** Anyone can read it, so do not list secret paths in it. - **User agents can be spoofed.** If you need to verify a crawler, check the provider's published IP ranges where available. For anything that must stay private, use authentication. Use your server logs to confirm crawlers behave as you expect; our post on [measuring visibility in AI answers](https://aispeedforce.com/blog/measuring-visibility-in-ai-answers/) shows how. Before you publish your rules ## Next step The free [agent-readiness scan](https://aispeedforce.com/agent-ready/) checks whether your robots.txt names the AI crawlers, declares Content Signals and points to a sitemap. To see how these files fit together, read [llms.txt explained](https://aispeedforce.com/blog/llms-txt-explained/) and the [agent readiness checklist](https://aispeedforce.com/blog/agent-readiness-checklist/), or ask us about our [SEO service](https://aispeedforce.com/services/seo/). ## Questions about this topic ### If I block GPTBot, will my site disappear from ChatGPT answers? Not necessarily. OpenAI documents GPTBot for training and OAI-SearchBot for search features as separate crawlers, so you can block one and allow the other. ### Does blocking Google-Extended remove my site from Google Search? No. Google documents Google-Extended as a control over use for its generative AI models, separate from Googlebot, which handles Search indexing. ### Is robots.txt enough to stop all AI crawling? No. robots.txt is a voluntary protocol. Reputable providers follow it, but it is not access control. Use authentication or server rules for content that must stay private. ## Related posts - [llms.txt explained: what it is and how to write one](https://aispeedforce.com/blog/llms-txt-explained/): SEO 23 September 2026. llms.txt is a short Markdown file at your site root that tells language models what your site is and where the important pages are. It is a proposal, not a standard, and it is cheap to publish. - [Content Signals explained: search, ai-input and ai-train in robots.txt](https://aispeedforce.com/blog/content-signals-explained/): SEO 23 September 2026. Content Signals add one line to robots.txt that says how your content may be used once it is fetched: for search, as input to AI answers, or for model training. - [Lighthouse's agentic browsing audit: what it checks and how to pass](https://aispeedforce.com/blog/lighthouse-agentic-browsing-audit/): SEO 23 September 2026. Recent Lighthouse versions add an Agentic Browsing category that checks how well AI agents can read and use a page. Here is what it looks at and how to fix what it finds. - [All posts](https://aispeedforce.com/blog/) - [More on SEO](https://aispeedforce.com/blog/category/seo/) - [Agent-readiness scan](https://aispeedforce.com/agent-ready/) ## See how your own site scores The free agent-readiness scan checks the files, headers and endpoints this blog writes about, and tells you what to fix first.