What 500 of the biggest domains do about AI crawlers, and what it means for a site your size

Five hundred of the largest domains on the web, asked one question. Does your robots.txt allow the crawlers that are now reading pages on behalf of a language model? The full research page, with the per-domain rows and the method statement, is at enterramon.com. This is what the numbers mean if you run a business site rather than one of those five hundred.

The sample is smaller than it sounds

124 of the 500 have no address at all. They are infrastructure names such as akamai.net and cloudfront.net, which were never meant to be visited, so they cannot have an opinion about crawlers. That leaves 376 real websites, and everything below is measured against those.

That second row is worth a sentence. Forty sites would not serve their own robots.txt to a request that identified itself, which means a firewall is refusing a file whose entire purpose is to be read by machines. From the outside, a site that blocks courteous crawlers and a site that blocks everyone look identical, and both are invisible to the tools their owners use to check.

Which crawlers get blocked

Thirty percent block at least one of the seven and 11% block all seven. The roll of the latter is mostly what you would predict from the copyright argument, publishers and social platforms, with a few surprises in both directions.

Three percent publish a Content-Signal

Cloudflare proposed the line in 2025 as a way to state training permission inside robots.txt, and almost nobody has taken it up. Three percent of 264 is about eight domains. That is not a reason to skip it, because the cost is one line and the alternative is being read as undecided.

What this means for a site your size

A small business site is not in a copyright fight, and copying the behaviour of the largest publishers would mean vanishing from the answers people are now getting from a model. The useful question is narrower. Which crawlers do you want reading your pages, and does your file say so on purpose?

Start by reading your own file rather than assuming it. Our robots.txt check tests real addresses against the rules the way a crawler applies them, and it names the AI agents separately, so you can see what each one is allowed to do.

Then decide, in one line each:

• Search crawlers should read everything you want indexed.
• Answer engines should read the pages you want quoted, with your name on the quote.
• Training crawlers are a policy choice, and either answer is defensible when it is written down.

How the numbers were produced

The top 500 entries of the Tranco list, list PYK6J built 29 October 2026, one crawl on 30 October 2026. Each robots.txt was fetched once and evaluated per agent using the group matching and longest match precedence in RFC 9309. A crawler is blocked when the rules that apply to it disallow the root. Requests carried a browser user agent with our own token appended, timeouts were twelve seconds, redirects were followed at most once, and a site that refused or failed is recorded as refused rather than retried until it agreed. Seven agents were counted, and the findings are stated over those seven rather than backfilled with the fifteen the tester now asks.

Use it, and how to credit it

The per-domain rows are downloadable as CSV from the research page, so the arithmetic can be checked rather than believed. The dataset is published under CC BY 4.0, which means you may copy it, chart it, quote it and republish it without asking us.

If you cite it, credit it as Enterramon, "What 500 of the Biggest Domains Do About AI Crawlers", 30 October 2026 and link the research page. Attribution is what the licence asks for. The link is what lets a reader check the method instead of taking the number on trust.

The CSV ships with a plain text README that repeats the column definitions, the denominators and this credit line, so the terms travel with the file rather than staying on the page somebody downloaded it from.