SEO
Technical and on-page SEO for a web that search engines and AI crawlers read side by side.
robots.txt for AI crawlers: GPTBot, ClaudeBot and the rest
AI providers run separate crawlers for training, for search and for fetching a page a user asked about. robots.txt lets you treat each one differently. Here is who is who and how to write the rules.Read the postllms.txt explained: what it is and how to write one
llms.txt is a short Markdown file at your site root that tells language models what your site is and where the important pages are. It is a proposal, not a standard, and it is cheap to publish.Read the postLighthouse's agentic browsing audit: what it checks and how to pass
Recent Lighthouse versions add an Agentic Browsing category that checks how well AI agents can read and use a page. Here is what it looks at and how to fix what it finds.Read the postJSON-LD schema that matters: the types worth adding and how to link them
A short list of schema.org types covers most business sites. Link them into one graph with @id, keep them true to the visible page, and validate before you ship.Read the postContent Signals explained: search, ai-input and ai-train in robots.txt
Content Signals add one line to robots.txt that says how your content may be used once it is fetched: for search, as input to AI answers, or for model training.Read the postSee how your own site scores
The free agent-readiness scan checks the files, headers and endpoints this blog writes about, and tells you what to fix first.