# Content Signals explained: search, ai-input and ai-train in robots.txt > What Content Signals are, what search, ai-input and ai-train mean, where the line goes in robots.txt, and why it states a preference rather than a rule. Canonical page: https://aispeedforce.com/blog/content-signals-explained/ Last updated: 2026-09-23 Content Signals add one line to robots.txt that says how your content may be used once it is fetched: for search, as input to AI answers, or for model training. [SEO](https://aispeedforce.com/blog/category/seo/) Published 23 September 2026 5 min read By AI SpeedForce Content Signals are a way to say, in your robots.txt file, how your content may be used after a crawler fetches it. There are three uses: `search`, `ai-input` and `ai-train`, and you set each one to yes or no. They express a preference. They do not block anything on their own. That distinction matters. For decades robots.txt has answered one question: may this crawler fetch this URL? AI changed what happens after the fetch. The same page might feed a search index, be quoted inside an AI answer, or end up in a training set. Content Signals give you a way to speak to those uses separately, instead of treating every crawler as all or nothing. ## The three signals The vocabulary is small on purpose. Each signal names one kind of use. | Signal | Covers | Example value | |---|---|---| | search | Building a search index and showing links and snippets | search=yes | | ai-input | Using content in an AI answer, such as grounding or RAG | ai-input=yes | | ai-train | Training or fine-tuning AI models | ai-train=no | The three signals - **search** covers building a search index and returning links and short excerpts to people. This is classic search engine behavior. - **ai-input** covers using your content as input to an AI system that answers a question right now, for example retrieval-augmented generation or grounding an assistant's reply in your page. - **ai-train** covers training or fine-tuning AI models on your content. The three are independent. A publisher might welcome search and AI answers that cite them, but not want their writing used to train a model. Another might be happy with all three. The point is that you can now say which. ## Where the line goes in robots.txt A Content-Signal line sits inside a normal robots.txt group, next to the Allow and Disallow rules. The proposal at [contentsignals.org](https://contentsignals.org/) shows it placed with the User-agent it applies to: `User-agent: * Content-Signal: search=yes,ai-input=yes,ai-train=no Allow: /` Two practical notes from putting this into production on our own site: - **Put it directly under the User-agent line.** Some checkers only recognize the signal when it sits at the top of its group. Placing it there costs nothing and avoids a false "missing" result. - **Repeat it in every group.** If you have a separate group for named AI crawlers and another for everyone else, each group needs its own Content-Signal line. A crawler reads only the group that matches it. Where the signal sits Comments are fine for humans reading the file, but keep them on lines that start with `#`. Some parsers are strict, and an unrecognized directive can make a validator flag the whole file. We learned that with a non-standard line of our own and removed it. ## A preference, not enforcement This is the part people most often misread. A Content-Signal line does not stop anyone. robots.txt itself has always been voluntary: well-behaved crawlers follow it, and nothing technical forces the rest. Content Signals inherit that. They tell a crawler operator what you want. Whether the operator honors it is their decision and their responsibility. So use each tool for what it does: - To **stop fetching**, use Disallow rules for the crawlers you name, or block at the server or CDN. - To **state how fetched content may be used**, use Content Signals. - To **make the terms binding**, put them in your published terms of use. A signal is evidence of your intent; your terms are the agreement. Our guide to [robots.txt for AI crawlers](https://aispeedforce.com/blog/robots-txt-for-ai-crawlers/) covers the access side: which user agents exist, what each is for, and how to allow search while limiting training. ## The standards picture Content Signals began as a published proposal with its own site and a short specification. In parallel, the IETF formed the AI Preferences working group (aipref) to define a shared vocabulary for expressing how content may be used by AI systems, and how that vocabulary attaches to content, for example through robots.txt or HTTP headers. Its documents are still drafts. What that means for you today: the three-signal line is simple, readable and already recognized by agent-readiness checkers. The exact standard vocabulary may still change as the IETF work finishes. Adopting the current line now is low effort, and you can adjust the wording later if the final vocabulary differs. Watch the working group's drafts on [datatracker.ietf.org](https://datatracker.ietf.org/) rather than second-hand summaries. ## Choosing your values There is no universally right setting. Think about each use in terms of what you get back. - **search=yes** is the default choice for almost every public business site. Search sends visitors. - **ai-input** decides whether AI assistants can use your pages when answering. If you want to be the source an assistant cites, this should be yes. That is the whole idea behind [answer engine optimization](https://aispeedforce.com/blog/what-is-answer-engine-optimization/). - **ai-train** is the real decision. Training does not send traffic back in any direct way. Some businesses allow it because they want models to know their product; others decline because their content is their product. On aispeedforce.com we currently state all three as yes, because we want assistants and agents to understand and describe our work accurately. We made that a deliberate choice and wrote it into our [site policies](https://aispeedforce.com/cookie-policy/) and robots.txt comments, so anyone reading the file knows it was intended. Before you publish your signals ## How to check yours Open your robots.txt in a browser and read it as a crawler would. Every group should have one Content-Signal line near the top. Then run the free [agent-readiness scan](https://aispeedforce.com/agent-ready/), which reads your robots.txt, reports whether Content Signals are present and valid, and lists the AI crawlers you name explicitly. If you are also publishing an [llms.txt](https://aispeedforce.com/blog/llms-txt-explained/) file, the two work together: llms.txt tells an assistant what is worth reading, and the signal tells it what it may do with what it reads. ## Next step Add one Content-Signal line to each robots.txt group, with a value you chose on purpose for each of the three uses. Then [scan your site](https://aispeedforce.com/agent-ready/) to confirm it is detected, along with the other files agents look for. ## Questions about this topic ### Do Content Signals block crawlers? No. They state how you want content used after it is fetched. Blocking is still done with Disallow rules or at the server. A crawler that ignores the signal is not stopped by it. ### What happens if I leave a signal out? Leaving a signal out expresses no preference for that use. It is neither a yes nor a no, so state all three if you have a view on each. ### Is Content Signals an official standard? It is a published proposal, and related vocabulary is being discussed in the IETF AI Preferences working group, whose documents are still drafts. Treat it as a clear statement of intent, not a legal control. ## Related posts - [robots.txt for AI crawlers: GPTBot, ClaudeBot and the rest](https://aispeedforce.com/blog/robots-txt-for-ai-crawlers/): SEO 23 September 2026. AI providers run separate crawlers for training, for search and for fetching a page a user asked about. robots.txt lets you treat each one differently. Here is who is who and how to write the rules. - [llms.txt explained: what it is and how to write one](https://aispeedforce.com/blog/llms-txt-explained/): SEO 23 September 2026. llms.txt is a short Markdown file at your site root that tells language models what your site is and where the important pages are. It is a proposal, not a standard, and it is cheap to publish. - [Lighthouse's agentic browsing audit: what it checks and how to pass](https://aispeedforce.com/blog/lighthouse-agentic-browsing-audit/): SEO 23 September 2026. Recent Lighthouse versions add an Agentic Browsing category that checks how well AI agents can read and use a page. Here is what it looks at and how to fix what it finds. - [All posts](https://aispeedforce.com/blog/) - [More on SEO](https://aispeedforce.com/blog/category/seo/) - [Agent-readiness scan](https://aispeedforce.com/agent-ready/) ## See how your own site scores The free agent-readiness scan checks the files, headers and endpoints this blog writes about, and tells you what to fix first.