Content Signals are a way to say, in your robots.txt file, how your content may be used after a crawler fetches it. There are three uses: search, ai-input and ai-train, and you set each one to yes or no. They express a preference. They do not block anything on their own.
That distinction matters. For decades robots.txt has answered one question: may this crawler fetch this URL? AI changed what happens after the fetch. The same page might feed a search index, be quoted inside an AI answer, or end up in a training set. Content Signals give you a way to speak to those uses separately, instead of treating every crawler as all or nothing.
The three signals
The vocabulary is small on purpose. Each signal names one kind of use.
| Signal | Covers | Example value |
|---|---|---|
| search | Building a search index and showing links and snippets | search=yes |
| ai-input | Using content in an AI answer, such as grounding or RAG | ai-input=yes |
| ai-train | Training or fine-tuning AI models | ai-train=no |
- search covers building a search index and returning links and short excerpts to people. This is classic search engine behavior.
- ai-input covers using your content as input to an AI system that answers a question right now, for example retrieval-augmented generation or grounding an assistant's reply in your page.
- ai-train covers training or fine-tuning AI models on your content.
The three are independent. A publisher might welcome search and AI answers that cite them, but not want their writing used to train a model. Another might be happy with all three. The point is that you can now say which.
Where the line goes in robots.txt
A Content-Signal line sits inside a normal robots.txt group, next to the Allow and Disallow rules. The proposal at contentsignals.org shows it placed with the User-agent it applies to:
User-agent: *
Content-Signal: search=yes,ai-input=yes,ai-train=no
Allow: /Two practical notes from putting this into production on our own site:
- Put it directly under the User-agent line. Some checkers only recognize the signal when it sits at the top of its group. Placing it there costs nothing and avoids a false "missing" result.
- Repeat it in every group. If you have a separate group for named AI crawlers and another for everyone else, each group needs its own Content-Signal line. A crawler reads only the group that matches it.
Comments are fine for humans reading the file, but keep them on lines that start with #. Some parsers are strict, and an unrecognized directive can make a validator flag the whole file. We learned that with a non-standard line of our own and removed it.
A preference, not enforcement
This is the part people most often misread. A Content-Signal line does not stop anyone. robots.txt itself has always been voluntary: well-behaved crawlers follow it, and nothing technical forces the rest. Content Signals inherit that. They tell a crawler operator what you want. Whether the operator honors it is their decision and their responsibility.
So use each tool for what it does:
- To stop fetching, use Disallow rules for the crawlers you name, or block at the server or CDN.
- To state how fetched content may be used, use Content Signals.
- To make the terms binding, put them in your published terms of use. A signal is evidence of your intent; your terms are the agreement.
Our guide to robots.txt for AI crawlers covers the access side: which user agents exist, what each is for, and how to allow search while limiting training.
The standards picture
Content Signals began as a published proposal with its own site and a short specification. In parallel, the IETF formed the AI Preferences working group (aipref) to define a shared vocabulary for expressing how content may be used by AI systems, and how that vocabulary attaches to content, for example through robots.txt or HTTP headers. Its documents are still drafts.
What that means for you today: the three-signal line is simple, readable and already recognized by agent-readiness checkers. The exact standard vocabulary may still change as the IETF work finishes. Adopting the current line now is low effort, and you can adjust the wording later if the final vocabulary differs. Watch the working group's drafts on datatracker.ietf.org rather than second-hand summaries.
Choosing your values
There is no universally right setting. Think about each use in terms of what you get back.
- search=yes is the default choice for almost every public business site. Search sends visitors.
- ai-input decides whether AI assistants can use your pages when answering. If you want to be the source an assistant cites, this should be yes. That is the whole idea behind answer engine optimization.
- ai-train is the real decision. Training does not send traffic back in any direct way. Some businesses allow it because they want models to know their product; others decline because their content is their product.
On aispeedforce.com we currently state all three as yes, because we want assistants and agents to understand and describe our work accurately. We made that a deliberate choice and wrote it into our site policies and robots.txt comments, so anyone reading the file knows it was intended.
How to check yours
Open your robots.txt in a browser and read it as a crawler would. Every group should have one Content-Signal line near the top. Then run the free agent-readiness scan, which reads your robots.txt, reports whether Content Signals are present and valid, and lists the AI crawlers you name explicitly.
If you are also publishing an llms.txt file, the two work together: llms.txt tells an assistant what is worth reading, and the signal tells it what it may do with what it reads.
Next step
Add one Content-Signal line to each robots.txt group, with a value you chose on purpose for each of the three uses. Then scan your site to confirm it is detected, along with the other files agents look for.