Check whether AI can read your site, and enable it if it cannot

Whether an AI assistant can read your site is not one question. It is five, and only the first one is robots.txt. Most advice stops at "allow GPTBot" and leaves the other four unexamined, which is why a site can be technically open to every agent and still hand them very little.

The robots.txt checker runs all five and prints a score with the finding behind each. This is what they are, in the order they matter.

One: may the agents fetch at all

Every AI crawler carries a user agent, and each is matched against the file's groups. The trap is a rule that looks permissive and is not: a User-agent: * disallow alongside a named GPTBot group that never authorises it, or a path rule that blocks the assistant's fetcher while leaving the trainer open.

The checker reports each agent separately - allowed, blocked, or blocked because of the wildcard group - and names the line that decided it. If you want the group-by-group reasoning, that is covered in which AI crawlers your robots.txt allows.

Two: is there an llms.txt

llms.txt is a proposed convention, not a standard: a Markdown file at the site root that tells a language model what the site is and which pages matter. Nothing enforces it and no crawler is obliged to read it, which is why it is worth being honest about - it is a signpost, not a permission.

A useful one is short and factual:

# EnterraHost

> Domains, business email and hosting, plus free network and SEO tools.

## Tools
- [Robots.txt checker](https://enterrahost.com/tools/robots-txt-tester/): tests a URL against the live file, per agent.
- [SPF, DKIM and DMARC checker](https://enterrahost.com/tools/spf-dkim-dmarc-checker/): reads the three records and explains each verdict.

## Support
- [Support articles](https://enterrahost.com/support/): how-to guides for mail, DNS and hosting.

What makes it worth writing is the second half of each line: the description is the part a model can quote. A list of bare URLs gives it nothing to say about you.

Three: is the page readable as Markdown

This is the check most sites fail without knowing. An agent that has fetched your HTML still has to parse navigation, cookie banners, footers and scripts to find the article. Markdown, or a clean text twin, is cheaper to read and survives being quoted.

Two ways to serve it, in order of effort:

A .md twin next to the page: /support/articles/foo/ also answering at /support/articles/foo/index.md. Cheap to generate, easy to link from llms.txt, and it can be a build step rather than an editor's job.
Content negotiation: return Markdown when the request carries Accept: text/markdown, HTML otherwise. Better behaved, more work, and it depends on the agent asking.

The checker reports whether a Markdown twin answered at all, not whether it is good.

Four: are those URLs in the sitemap

A llms.txt that lists pages nobody can find is a dead end. If the pages you are advertising to models are not in sitemap.xml, the file and the sitemap disagree about what your site contains - and the sitemap is the one crawlers already read.

Five: what the file may say about use, not only access

robots.txt answers "may you fetch this". It does not answer "may you train on it", and pretending it does is how sites end up with a policy nobody can enforce. Structured statements about use - a content licence, a machine-readable declaration, a Content-Signal-style header - are advisory in every practical sense, and the checker reports them as advisory rather than as pass or fail.

Reading the score

The score is only emitted when every check could be answered. If a site blocks the tester itself, or the file cannot be fetched, the score is withheld and the findings are shown instead - a number built on a guess is worse than no number, because it gets screenshotted and compared.

A high score means an agent can fetch and read what you publish. It does not mean anything will cite you. Being fetchable is a precondition for being quoted, not a promise of it, and any tool that claims otherwise is selling you the second thing while measuring the first.

What to do, in order

Decide which agents you actually want. Training crawlers and answer-engine fetchers are different, and blocking one does not block the other.
Allow those in robots.txt, and check the file rather than reading it - the wildcard trap is invisible to the eye.
Publish llms.txt with descriptions, not just links.
Serve Markdown or a text twin for the pages that matter.
Put those URLs in the sitemap.

Then re-run the check. None of it takes an afternoon, and every step is verifiable from outside your site, which is the only kind of claim worth making about visibility.