Make your site readable by AI agents, and know where to stop

An automated agent is a program that fetches a page, reads it, and uses what it finds to answer a question or take an action. It is not a browser. Nothing is looking at a layout, clicking a link or scrolling past a banner. Something asked a question, an agent went looking for an answer, and your page either supplied one or did not.

That has become part of how people find things. Someone asks an assistant what a service costs or whether a tool does a particular job, the assistant reads a handful of pages, and it answers with the two or three it could parse cleanly. A page that renders beautifully in a browser and arrives as a hundred kilobytes of nested markup can lose that contest to a plainer page that arrives as text.

The difference from search matters here. A search crawler indexes a URL and returns a link, and the person decides what to read. An agent reads the page itself and returns an answer, usually naming where the answer came from. Being the page it names is the whole objective.

This page is what we changed on our own site to make that work, the two faults we hit on the way, and the part we deliberately left undone.

What an agent does with a page

The mechanics are simple enough to describe without a product in the room.

An agent fetches the URL. It parses the response. It discards whatever carries no meaning, which on a typical marketing page is most of the bytes. It keeps the text, often broken into pieces small enough to quote.

Every step of that is easier when the response is text and harder when it is markup. A browser needs the markup because it has to paint boxes, position a navigation bar and load a font. An agent needs none of that, and everything it has to strip is attention it was going to spend on the answer.

The second half of the problem is permission. Where a crawler may go has been settled for decades. What a reader may do with the text afterwards is a newer question, and it needs a newer answer.

How to tell where you stand

Several tools now grade a site on this. We used Cloudflare's URL Scanner, which has an "Agent readiness" option, because this site already runs behind Cloudflare and the check costs nothing extra.

It grades five groups and places the result on a four level scale. Our site started at 14 out of 100, which is level 1, and now sits at 27, which is level 3.

The levels run from a basic web presence at level 1, through machine readable declarations at level 2, to Agent-Readable at level 3. Level 4 is Agent-Integrated, where the action and payment endpoints arrive.

Two of those groups, protocol discovery and commerce, are about transacting with an agent rather than being read by one. They are dealt with further down, because deciding not to build them is the most useful part of this page.

The three changes that took us to level 3

Level 1 to level 3 is three changes, on a site that already has a working sitemap and robots.txt.

Say what a reader may do with what it reads

robots.txt answers where a crawler may go. It has no vocabulary for what happens after the text is fetched, which is the question that matters once a model can keep it.

Content-Signal is a line in robots.txt that answers the second question.

Content-Signal: search=yes, ai-input=yes, ai-train=no

Three settings on one line, and each is a separate decision.

• search=yes allows the page to be indexed and linked to.
• ai-input=yes allows a reader to use the text to answer a question, which is the quoting behavior most sites want.
• ai-train=no withholds it from training.

The last one is the interesting one, and a business should decide it on purpose. Training takes the text permanently and returns nothing to the page it came from. A model trained this year quotes the price list it read this year, and when the price changes the model has no way to know and no duty to correct itself. Answering from the live page has none of that problem, which is why one is allowed here and the other is not.

A content signal is a declaration rather than an enforcement. A crawler that ignores it is not blocked by it, the way a Disallow line blocks a crawler. It states an intent that well behaved readers act on, and it costs a line.

The robots.txt tester reports every Content-Signal a file carries and what each setting permits, so you can see whether yours says anything at all.

Serve the page as Markdown when it is asked for

An agent can parse HTML. It would rather not, and on a heavy page the difference is large enough to matter.

Our homepage is 101 KB of HTML. The same page as Markdown is 12 KB. Almost all of that difference is 45 inline SVGs, nine scripts, the site header and the footer. None of it carries meaning, and all of it exists so a browser can paint the page.

The negotiation runs on the Accept header, which is how any client already says what it can handle. A caller sending text/markdown at a higher weight than text/html gets the page as Markdown. The weight is the quality value a client puts on each format it will accept, and it is how the same program can say "give me Markdown, but HTML will do". A browser never matches, because no browser asks for Markdown, and where the two weigh the same the HTML wins.

Strictness is the part that keeps it safe. Only GET and HEAD, the two request methods that fetch without changing anything, are converted. Only a successful response is converted. Only a response whose content type is HTML, or absent, is converted. So a data endpoint still returns data, a redirect still redirects, and a form submission is never rewritten.

That list is not decoration. This site has a signed checkout handoff, and a request that changes state must not be transformed by something that only meant to read a page.

You can watch it happen on the tools index. Ask for that URL with Accept: text/markdown and the same page comes back as text.

Point at the files that describe the site

A sitemap lists URLs for a search crawler. llms.txt is a newer convention that does something different. It is a plain text file, written in Markdown, describing what the site is and which pages are worth reading. Ours sits at /llms.txt, and a longer /llms-full.txt holds the text of every indexable page for a reader that wants everything at once.

Neither file is a standard, and nothing is obliged to read them. The point is that a reader which looks for them finds a short and accurate description rather than having to infer one from a crawl.

The way to advertise them is a Link header, one of the lines a server sends before the page body that a browser never shows you.

Link: </llms.txt>; rel="describedby llms-txt", </sitemap.xml>; rel="sitemap"

That rel value is the interesting part. It names the type of relationship between the page and the file, and only the registered names mean anything to software that has not been told about your convention.

The first version of ours used rel="llms-txt", which looks like the obvious choice and scored red. The scanner had parsed the header correctly and found nothing useful in it, because llms-txt is not a registered name. It meant something to us and nothing to anyone else.

describedby is registered, in RFC 6892, and it is the honest fit, because an llms.txt file is a machine readable description of the site. Both values now ride on one header, and that header is sent on the HTML and Markdown responses alike, because it describes the resource rather than either representation of it.

The sitemap validator reads that header on your root document, checks whether an llms.txt answers, and reports it when a header promises a file that is not there.

Where we stopped, and why

The remaining points are protocol discovery, which is nine checks, and commerce, which is five. All fourteen describe a way for an agent to do something on a site rather than read it, and the names mean very little until they are grouped by what they are for.

Telling an agent what it can ask for. An API catalog is a machine readable list of the endpoints a site offers. It answers "what can this site do" without the agent having to guess, or read documentation written for a person.

Telling an agent how to sign in. OAuth and OpenID discovery are the established way for a program to find a service's login flow, its identity provider and the permissions it can request, instead of a developer wiring those details in by hand. auth.md is a newer proposal for describing the same thing in one file a person can also read.

Telling an agent about your tools. The Model Context Protocol is a common way for an assistant to call into another system, and a server card is the short file that describes what that system offers. The Agent to Agent protocol does the same job between two agents, and its agent card says what one agent can ask another to do. A skills index is the same idea once more, as a list of named abilities. WebMCP is the browser version, so an assistant driving a page can use the buttons and fields rather than guessing at what it sees.

Taking payment. Five protocols cover the same ground under five names. x402, the Model Payments Protocol, the Universal Commerce Protocol, the Agentic Commerce Protocol and the Agent Payments Protocol all describe a way for an agent to pay for something without a person typing card details.

Every one of those advertises an action an agent can take on your behalf. That is the right thing to build when your site is a platform that agents are meant to transact against, and there are real cases where it pays. Most business sites and online stores are not that, and adding the endpoints does not make them that. It adds a surface.

For us the answer is narrower than a preference. Our cart is an envelope signed with a shared secret, scoped to a single endpoint, carrying a whitelist of allowed actions, with the owner resolved from a cookie and never from the payload. That design is why a crafted request cannot drive somebody else's cart. An endpoint that let an agent add to a cart, or pay for one, would trade a security property for a badge.

DNS for AI Discovery is red for the same reason and looks more innocent than it is. It reads like a DNS record, and per draft-mozleywilliams-dnsop-dnsaid-02 the record carries two parameters, one naming the protocol to speak and one pointing at a capability descriptor such as /.well-known/mcp/server-card.json. It is the DNS layer of the same story, and it would advertise endpoints this site does not run.

A future scan showing 27 is not a to-do list.

Two things that went wrong

Neither is specific to agents, and both are the kind of fault that never produces an error a visitor can see.

A Vary header that nginx was overwriting

The first version of the Markdown negotiation sent Vary: Accept, which is the textbook way to tell a cache that one URL has two representations. This host sets gzip_vary on globally, so nginx appends its own Vary: Accept-Encoding to every response it compresses. PHP's header() replaces by default, so the two fought and nginx won. The HTML path went out with Vary: Accept-Encoding alone, and a shared cache was then free to answer a browser with Markdown.

It was invisible from the Markdown side, where the header survived because there was nothing to disagree with. The fix is to send the combined value, Vary: Accept, Accept-Encoding, before the early return, so the HTML path gets it as well.

The same shape appeared again in a cache rule on this host, which is the reason it earns the space here. A Cloudflare rule had been set to ignore the origin's cache control header and hold responses for a day. This site answers every page with no-cache, because prices are read live from the billing system at render time, so the rule overrode it and every page carrying a price was served from a twenty hour old copy. The rule had to be changed to respect the origin before the pages were fresh again. A cache that has learned the wrong answer does not report the mistake to anyone.

A Permissions-Policy directive that is not one

The header carried usb=(), which is not a directive in the specification, so no browser has ever had a setting to switch off. The browser logged the entry as unrecognized on every page and ignored it. Removing the line changed nothing a visitor could see and stopped the console filling with a complaint that was correct.

What to do on your own site

In the order that costs least for what it returns.

1. Make sure robots.txt and a sitemap exist and are correct, because almost nothing else counts without them.
2. Add a Content-Signal line to robots.txt, and decide the training answer deliberately rather than by inheritance.
3. Send a Link header naming your llms.txt and your sitemap, using describedby rather than an invented name.
4. If you can serve Markdown, serve it strictly, and leave anything with a side effect alone.
5. Read the protocol and commerce groups, and decide whether you are a platform. Most sites are not, and leaving them red is the honest answer.

Three of those five are one line of configuration. The other two are decisions not to build something.