Skip to content
EnterraHost
Products
Tools
Hosting Email Support Client Area Find a domain
Easy utilities

Robots.txt Check

Run a robots.txt check on any site. It reads the file, reports what is malformed, and tests real addresses against the rules the way a crawler applies them.

A robots.txt check tells you two things. Whether the file is readable and correctly formed, and whether the addresses you care about are actually allowed. A file can be valid and still block the pages you want crawled, which is why the file is only half the answer.

Reads the robots.txt at the site root, and checks how AI crawlers are treated in the AI visibility tab. Public addresses only, and a robots.txt that redirects is followed up to three times, with every hop checked before it is fetched.

Email me this result

The same result, sent to your inbox.

We use the address to send this result. It is not stored after the message goes out, and nothing is added to a list unless you tick the box.

Need somewhere to put the site? Hosting from $4.95 a month, with free DNS on every domain you own.

The part people get wrong

Disallow: / blocks everything, including the page you wanted found.

Most people who search for "robots txt disallow" have one of three rules open and want to know what it does. Disallow: / blocks the whole site. Disallow: with nothing after it blocks nothing at all. Disallow: /private/ blocks one directory and every path underneath it. A crawler matches from the start of the path, so the leading slash matters and a rule without one is ignored.

None of those three removes a page from Google. A disallowed URL is not crawled, so it is never read, so a noindex on it is never seen - and a page linked from somewhere else can still be indexed on the strength of that link. Blocking and removing are different jobs: use robots.txt to stop a crawler wasting time, and use a noindex header or tag when the goal is to keep a page out of the index. Test the exact URL below with the crawler you care about - the answer depends on which group in the file applies to that agent, not on the file as a whole.

Robots.txt controls crawling, not indexing.

This is the single most expensive misunderstanding in the file. A URL you disallow is not removed from search, and can still rank. A crawler that is blocked never reads the page, so it never sees a noindex instruction, and a page linked from elsewhere can be indexed on the strength of that link alone.

To block crawling

Use robots.txt. This is what it is for, and it is the right tool for keeping a crawler out of paths it wastes time on, such as internal search results or faceted URLs.

To keep a page out of the index

Use a noindex robots meta tag or the equivalent HTTP header, and let the page be crawled so the instruction is actually read. Blocking the crawl and asking for noindex at the same time cancels the second request.

Matching

Three rules that decide the answer.

Prefix, not exact

Disallow /private also blocks /private-notes and /private/report. A path with no wildcard and no end anchor matches anything that starts with it, which is why a rule that looks tidy can be wider than intended.

Longest match wins

Allow and Disallow are not a first-match list. Every rule is tested and the most specific one, measured by pattern length, decides. A short Disallow cannot override a longer Allow.

Allow wins a tie

When an Allow and a Disallow match with the same pattern length, the allow rule wins. That is the behavior in RFC 9309 and it is the opposite of what most people expect when they write the two lines in the wrong order.

Permissions

The file can also say what may be done with what it reads.

For most of the protocol's life, a robots file answered a single question about where a crawler may go. It had no vocabulary for what happens after the page is fetched, and that gap matters now that a reader can keep the text.

Content-Signal is a line that answers the second question. It carries one or more settings written as name and value pairs, and each is a separate decision. This tool reports every one it finds and explains what it permits.

Content-Signal: search=yes, ai-input=yes, ai-train=no
  • search=yes allows the page to be indexed and linked to.
  • ai-input=yes allows the text to be read in order to answer a question, and quoted in the answer.
  • ai-train=no withholds it from training.

It is a declaration, not an enforcement. A crawler that ignores it is not blocked by it the way a Disallow line blocks one, so it states an intent rather than imposing one. A file that carries no Content-Signal is silent rather than permissive, and a reader that looks for one finds nothing to go on.

The tool also reports any Sitemap line and the address it declares, which is the same file the sitemap validator reads in full. Both are file level declarations rather than crawl rules, which is why neither is scoped to a user-agent.

Next

Once crawling is right, check what it finds.

A correct robots file does not tell you whether the pages behind it resolve. Neither does the sitemap it names, unless that file is valid too. Check either one next.

The full walkthrough is in how a robots.txt rule decides, including why a blocked page can still appear in search.

Questions

Questions about robots.txt.

Can I get the result by email?
Yes. Run the check, then press the paper plane beside the result and give it an address. The message carries the same figures the page shows, any table the result produced, and on the map tools the diagram itself as an attached vector file. Nothing is stored unless you tick the box asking to hear about new tools and trials, and if you do, the message carries a link that removes you again.
What does Disallow: / do?
It tells every crawler in that group to stay out of the whole site. The path is matched from the first character, so Disallow: / with a single slash blocks everything, while Disallow: with nothing after it blocks nothing. It does not remove anything from Google: a blocked URL is never crawled, so a noindex behind it is never read, and a page linked from elsewhere can still appear in results. To keep a page out of the index, use a noindex header or meta tag, and let the crawler reach the page so it can see it.
Does robots.txt remove a page from Google?
No. It controls crawling, not indexing. A blocked URL can still appear in search results if other pages link to it, because the crawler never fetches the page to discover it says otherwise. If you need a page out of the index, the robots meta tag or an HTTP header is the tool that does it, and blocking the crawl can prevent that instruction from ever being seen.
How are robots.txt rules matched?
Rules follow RFC 9309. A pattern without a wildcard or an end anchor is a prefix match, so Disallow /private also covers /private-notes. An asterisk matches any run of characters and a trailing dollar sign anchors the end. The longest matching pattern wins, and when two patterns are the same length the allow rule wins.
Why does an unknown line in my file matter?
Because behavior has changed. RFC 9309 requires crawlers to ignore a line they do not recognise and keep reading the rest of the file. Older crawlers could stop parsing at the first unknown directive, which would silently discard everything after it. A typo is therefore worth fixing even though current crawlers cope.
What is a Content Signal in robots.txt?
A declaration about what may be done with the text after it has been fetched, which is a different question from whether a crawler may fetch it. It is written as a Content-Signal line carrying one or more name and value pairs. Three settings are in common use, search, ai-input and ai-train, and each is set to yes or no. Unlike a Disallow line it is not enforced by anything. A reader that chooses to honor it does, and one that does not is not stopped.
Should I set ai-train to no?
It depends on what you want from the text, and it is worth deciding rather than leaving to a default. Training keeps the text permanently and returns nothing to the page it came from, so a model trained this year quotes a price list it read this year and has no way to know when the price changes. Allowing a reader to answer a question from the live page does not have that problem, which is why a site can reasonably allow one and refuse the other. Setting it to no is a request rather than a block.
What does Disallow with no path mean?
An empty Disallow allows everything. It is the conventional way to write a group that sets a crawl delay or states a policy without blocking any path, and it appears in a great many files. An empty Allow has no meaning at all.
Does your checker follow redirects?
No, and deliberately. A redirect points at a second address that nothing has validated, so following one would let a site aim our server at an internal host. If a robots.txt redirects, the tool tells you and you can paste the destination instead.
Which crawler should I test as?
Test as the one you care about. Rules are matched per user-agent group, longest token first, falling back to the wildcard group. A file can block a specific crawler while allowing everyone else, so a single test as the wildcard can be misleading.
Is a robots.txt checker the same as a tester?
The two words describe one job. Both fetch a robots.txt file and answer a single question about a specific URL. This one parses the file once and keeps the rules in the page, so testing ten paths against it costs one request rather than ten.
Do I need an Allow rule to let crawlers in?
No. A crawler that finds no matching Disallow line is allowed to fetch, so a file with no rules permits everything. An Allow line earns its place when it sits under a broader Disallow and has to carve out an exception, and the longest matching rule decides which of the two wins.
Should I disallow pagination?
Blocking paged URLs keeps a crawler out of the list pages, and the links on page one then point at addresses it cannot fetch. Anything reachable only through those links stops being discovered. If the pages carry their own canonicals, leaving them crawlable is usually the safer choice.