llms.txt is a Markdown file placed at the root of a website, at /llms.txt, that tells language models what the site is and where its most important pages are. It was proposed at llmstxt.org in 2024 as a simple way to give AI systems a clean guide instead of making them work it out from HTML. It is a proposal, not an official standard, and it takes less than an hour to publish.
This post explains what the proposal says, how to structure the file, how it relates to robots.txt and sitemaps, and what it will not do for you.
The problem it solves
Web pages are built for browsers. A typical page carries navigation, scripts, cookie banners, footers and layout markup around a few paragraphs of real content. A language model reading that page has to spend effort, and context space, working out what matters. It may also miss the pages that best explain your business, because nothing points to them.
llms.txt answers that with a short, curated guide written in Markdown, a format models read easily. It says: this is who we are, and these are the pages worth reading, with a line on each.
The structure the proposal describes
The proposal describes a specific order of sections:
- An H1 with the site or project name. This is the only required part.
- A blockquote with a short summary of what the site is, with the key facts a reader needs to understand the rest.
- Optional paragraphs or lists with more detail, without headings.
- H2 sections containing lists of links. Each item is a Markdown link, optionally followed by a colon and a short description.
- An optional section titled "Optional" for links that can be skipped when a shorter context is needed.
A small example:
# Example Store
> Example Store sells refurbished office furniture in the UK and ships within five working days.
## Products
- [Desks](https://example.com/desks.md): standing and fixed desks, with dimensions
- [Chairs](https://example.com/chairs.md): ergonomic chairs and warranty terms
## Policies
- [Delivery](https://example.com/delivery.md): areas, times and costs
- [Returns](https://example.com/returns.md): the 30-day returns processYou can see a full working file on this site at /llms.txt.
Markdown versions and llms-full.txt
The proposal also suggests offering a Markdown version of each page at the same URL with .md added. That gives models the page content without the HTML around it. Some sites serve Markdown on request instead, when a client asks for it with an Accept: text/markdown header; our post on Markdown for agents covers that approach.
A common companion is llms-full.txt: one file that contains the full text of the key pages, so a model can load everything in one request. It is useful for smaller sites and documentation. For large sites it can become too big to be practical, so keep it to what matters.
How it compares with robots.txt and sitemaps
| Purpose | Format | Audience | |
|---|---|---|---|
| robots.txt | What may be crawled | Plain text rules | All crawlers |
| sitemap.xml | Every URL to index | XML list | Search engines |
| llms.txt | What matters and why | Markdown | Language models and agents |
The three files do different jobs and do not replace each other:
- robots.txt is a set of rules about what crawlers may fetch. It is a standard, RFC 9309. See robots.txt for AI crawlers.
- sitemap.xml lists every URL you want indexed, for search engines.
- llms.txt is a curated guide to what matters, for language models and agents.
Keep them consistent. There is no point listing a page in llms.txt that robots.txt blocks for the crawler you hope will read it.
What llms.txt does not do
Be realistic about it:
- It is not a ranking signal you can rely on. No major search engine has said it uses llms.txt for ranking.
- It does not control access. It cannot stop anyone from reading or training on your content. Access rules belong in robots.txt, and usage preferences in Content Signals.
- It is not guaranteed to be read. Adoption varies by provider and tool.
So why publish one? Because it is cheap, it helps the tools and agents that do read it, and writing it forces a useful exercise: deciding which pages actually explain your business. Some documentation platforms and developer tools already generate or read it.
Where it lives and how it is served
Put the file at the root of your domain, so it is available at https://yourdomain.com/llms.txt. Serve it as plain text in UTF-8, keep the URL stable, and make sure robots.txt does not block it. If you also publish Markdown versions of pages, check that they are served as text and not as a file download, which some servers do by default for the .md extension.
Generate the file from the same source as your site where you can. A hand-edited llms.txt tends to fall behind the site: a page is renamed or removed and the link in the guide breaks. On this site the file is rebuilt from the page list every time the site is built, so it cannot point to a page that no longer exists.
Finally, link to it where agents might look. A Link header or a mention in your documentation helps tools find it, but the root location is the one the proposal defines.
Writing a good one
Write the summary for someone who has never heard of you. State facts, not slogans. Link the pages that answer the questions customers ask most, with a plain description of each. If some things are off limits, such as prices you do not publish, say so in the summary, so a model does not invent them.
Next step
The free agent-readiness scan checks whether your site has an llms.txt and whether it follows the expected shape, along with robots.txt, Markdown and structured data. For the wider picture of files agents look for, read the agent readiness checklist.