Agentic browsing is when an AI agent visits websites on a person's behalf: it finds pages, reads them and takes actions such as filling in a form or starting a checkout, instead of the person clicking through themselves. For a site owner, it means some of your visitors are now software working for a customer, and they read your site differently from a human.
This guide explains what these agents do, how they differ from people and from search crawlers, and the practical changes that make a site work well for them without making it worse for anyone else.
What an agent actually does on a site
An agent starts from a goal the person gives it, such as "find a plan that includes API access and tell me the monthly price" or "book the earliest appointment next week". It then works through the web much as a person would, only faster and with less patience for ambiguity.
Most agents repeat a short loop. They find a likely page, read it, decide what to do next, act, and check the result. Some run inside the person's own browser and can use the pages they are already signed in to. Others run on a server and fetch pages directly. Either way, the agent has to understand your page from its structure and text, because it cannot rely on visual cues the way a person does.
Good agents also pause before steps that cannot be undone, like paying, deleting or sending a message, and ask the person to confirm. That human confirmation step matters, and we come back to it below.
How agents differ from people and from crawlers
A person scans a layout, spots the big button and guesses from context. An agent reads the page's text and structure, and finds controls by their role and accessible name, the same information a screen reader uses. When a button is an unlabeled icon, or a form field has no label, a person can usually work it out. An agent often cannot.
| A person | An AI agent | |
|---|---|---|
| Reads the page by | Looking at the layout | Parsing text, structure and labels |
| Finds a button by | Its look and position | Its role and accessible name |
| Handles a vague label | Guesses from context | Often guesses wrong or stops |
| Prefers | A clear design | Clear structure and machine-readable files |
Agents are also different from search crawlers. A crawler such as Googlebot fetches pages to build an index that many people search later. An agent visits a handful of pages for one person and one task, right now. The two overlap, and the same site improvements help both, but the goals are not the same. Crawlers decide whether you are found. Agents decide whether the task gets done on your site or somewhere else.
Why it matters for your business
When an agent works on someone's behalf, your site competes on how easy it is to complete a task, not only on how it looks. If an agent cannot find your pricing, cannot tell which button submits the form, or gets stuck in a pop-up, it may report back that the information was unavailable, or move on to a competitor whose site it could use.
There is also the question of accuracy. An agent that misreads a page may repeat the mistake to a customer. Clear structure and explicit facts reduce the chance that your prices, policies or product details are summarized wrongly.
None of this requires guessing how any one AI product works inside. The practical changes rest on published standards and on plain good web practice, which is why they tend to hold up as the products change.
What agents look for on a site
Agents use whatever makes a page easiest to understand. The more of these a site offers, the less the agent has to guess.
- A guide to the site. An
llms.txtfile at the root gives language models a short Markdown summary of what the site is and where the important pages are. The format is described at llmstxt.org. - Clean text versions of pages. Serving a Markdown copy of each page, linked from the page head or returned when a client asks for
text/markdown, saves an agent from stripping menus and scripts out of your HTML. Our post on Markdown for agents shows how. - Structured data. JSON-LD from schema.org states facts such as your organization, products and FAQs in a form that does not need interpreting. See the JSON-LD types that matter.
- Labeled, predictable forms. Every field needs a real label, every button a clear name, and errors need to say what went wrong. We cover this in forms and checkouts agents can complete.
- Clear rules. robots.txt and Content Signals tell automated visitors what they may fetch and how your content may be used.
- Actions as tools. WebMCP, a draft web standard, lets a page offer its key actions to an in-browser agent as named tools, instead of making it drive the interface. Read WebMCP explained.
What to change first
Start with the changes that also help people, then add the files that are only for machines. A sensible order:
- Fix the basics of the page. One clear heading structure, real links and buttons rather than clickable boxes, labels on every form field, and text that is not locked inside images. This is standard accessibility work, and agents rely on the same signals as assistive technology.
- Publish llms.txt and a sitemap. Both are small files that tell any automated visitor what exists and what matters.
- Add structured data. Organization, WebSite and WebPage at least, plus Product, FAQPage or others where they genuinely apply.
- State your crawler preferences. Decide which AI crawlers you allow and publish Content Signals in robots.txt.
- Offer Markdown and, where it fits, tools. These are the steps that most reduce an agent's guesswork on a content-heavy or task-heavy site.
Keep a person in the loop for actions with consequences. If an agent can place an order or send a message through your site, the final step should show a clear summary and wait for the person to confirm. That protects your customers and your business from an agent that misunderstood the request.
Next step
If you want to see where your own site stands, run the free agent-readiness scan. It checks the files, headers and endpoints described above and explains how to fix each gap. If you would rather have it done for you, tell us about your site.