A robots.txt allow and disallow rule decides one thing only. May this crawler fetch this URL? It does not decide whether the URL can appear in search results, and it is not a way to remove a page. Those are different jobs with different tools, and confusing them is the most common mistake on the web.
You can fetch a robots file and test real paths against it without an account. This is how the decision is made.
What a robots.txt file is for
The file lives at the top level of a domain, at /robots.txt, and it tells crawlers which parts of the site they may fetch. Its purpose is to keep crawlers out of places they do not need to go, which protects your server from wasted requests and keeps your crawl budget pointed at pages you want indexed.
Google says this plainly in its own documentation. The file is used mainly to avoid overloading a site with requests, and it is not a mechanism for keeping a web page out of Google. Anything that claims otherwise is selling you a removal tool that does not remove anything.
How a robots.txt allow and disallow rule is chosen
A file is divided into groups. Each group starts with one or more user-agent lines naming the crawlers it applies to, and the allow and disallow lines that follow belong to that group until another user-agent line appears.
Three rules decide the outcome, and only the first two are well known.
The group is chosen by the most specific matching user agent. A crawler looks for the group that names it most precisely and ignores every other group, including the one for everyone. Google's own description of the specification puts it as finding the group with the most specific user agent that matches, with the rest ignored. So a site that writes a User-agent: * block and expects it to also cover Googlebot is wrong whenever a Googlebot group exists elsewhere in the file.
Inside the group, the longest match wins. RFC 9309, the standard that governs all of this, requires the most specific match to be used and defines most specific as the one with the most characters. A rule for /shop/private/ beats a rule for /shop/ every time, no matter which one appears first in the file. Order in the file is not precedence.
If an allow and a disallow match equally, allow wins. That tie-break is in the same section of the standard, and it is the reason an empty disallow line is safe. Disallow: with no path matches nothing and allows everything.
Two smaller details catch people out. A rule that appears before the first user-agent line belongs to no group and is ignored, so a file that opens with a bare disallow line blocks nothing at all. And wildcards are supported, with * standing for any run of characters and $ anchoring a match to the end of the path, though both are an extension to the original convention rather than part of the oldest implementations.
Disallow syntax, with examples
A disallow rule is a prefix match on the path, starting at the leading slash. It is not a regular
expression, there is no globbing beyond two characters, and everything after a # is a comment.
Most confusion comes from expecting it to behave like a pattern language when it is closer to "does the path
start with this".
Disallow: /blocks the whole site. A single forward slash matches every path.Disallow:with nothing after it blocks nothing. This is the correct way to say "no restrictions" for a group, notDisallow: /.Disallow: /private/blocks/private/and everything under it, but not/private-notes— the trailing slash is part of the match.Disallow: /searchblocks/search,/search/resultsand/searching, because it is a prefix and there is no trailing slash to stop it.Disallow: /*.pdf$blocks any path ending in.pdf:*matches any sequence,$anchors the end.Allow: /private/public/can sit alongsideDisallow: /private/. The more specific (longer) match wins, and an allow wins a tie of equal length.
Two rules of the format catch people repeatedly. Rules apply only within their user-agent group, so a
Disallow written under User-agent: * does not apply to a group that names
Googlebot explicitly — the most specific group that matches is the only one used. And a
* group does not automatically cover named crawlers when a named group exists, which is how a
site ends up allowing Googlebot while blocking everything else.
If a rule is meant to stop indexing rather than crawling, it cannot do that. A disallowed URL can still be indexed from links, it just cannot be fetched to be read. Test a specific path against the live file to see which rule wins, and the tool names the rule that matched rather than only saying "blocked".
The same check now runs per AI agent. The tester's AI visibility panel evaluates GPTBot, ClaudeBot,
PerplexityBot and the rest against the file separately, reports whether an llms.txt and a Markdown
twin exist, and scores the lot — so "is my site readable by AI" stops being a guess.
How to check and enable AI visibility
walks through what each finding means.
A blocked page can still appear in search
This is the part worth understanding properly, because it is where the money is spent on the wrong fix.
Blocking a URL tells a crawler it may not fetch it. It does not tell a search engine that the URL is unimportant. Google is explicit that a page blocked by robots.txt can still have its URL indexed if other pages link to it, and the crawler does that without ever visiting the page.
The trap that follows is worse. The usual advice for a page you want gone is to add a noindex tag. Adding one to a page that robots.txt already blocks achieves nothing, because the crawler is not allowed to fetch the page and therefore never reads the tag. You have blocked the only mechanism that could have removed it. Blocking and noindexing are opposites here, and using both gives you the worst outcome rather than the strongest one.
If the goal is removal, the order matters. Let the crawler fetch the page, serve a noindex tag, and block the URL in robots.txt only after the page has dropped out of the results. That is also why a robots.txt file is the wrong tool for anything sensitive. It is a public document, anybody can read it, and a disallow line is an announcement that the path exists.
What robots.txt will not do
It will not slow a crawler down on request. The crawl-delay field is not in the standard, and Google states that fields such as crawl-delay are not supported, so crawl rate is configured in a search console rather than in this file.
It will not secure anything either. A disallowed path is still a reachable path, crawlers are not the only things that read URLs, and this has never been an access control.
It also does nothing for rankings in the way people hope. There is no crawl budget stored somewhere that improves a page by keeping crawlers away from other pages. The file directs effort, and past that its effect on rankings is indirect.
The last limit is easy to hit and hard to notice. Google enforces a 500 KiB limit on the file, which is 512,000 bytes, and ignores everything past that point, so a giant generated file stops working quietly at the place it was most needed.
What the file can say about the text, not only the crawl
Every rule above answers one question, which is whether a crawler may fetch a path. The protocol had no vocabulary for what happens after the fetch, and for most of its life that did not matter because the only thing a crawler did with a page was index it.
It matters now that a reader can keep the text. Content-Signal is a line that answers the second question, and it is written as one or more settings in a name and value pair.
Content-Signal: search=yes, ai-input=yes, ai-train=no
Three settings are in common use. search governs whether the page may be indexed and returned as a result. ai-input governs whether the text may be read in order to answer a question, and quoted in the answer. ai-train governs whether it may be kept to train a model. Each is set to yes or no, and each is a separate decision.
The training setting is the one worth thinking about rather than copying. Training keeps the text permanently and returns nothing to the page it came from, so a model trained this year quotes the price list it read this year and has no way to know when the price changes. Answering a question from the live page does not have that problem, which is why a site can reasonably allow one and refuse the other.
None of it is enforced. A Disallow line is obeyed because ignoring it is detectable and the crawler has agreed to the protocol. A content signal is a declaration of intent, and a reader that ignores it is not blocked by it and cannot be. That is not a reason to leave it out, because a reader that looks for one and finds nothing has been told nothing, and silence is not the same as permission.
If a file carries no Content-Signal, the checker says so rather than reporting a fault. A missing declaration is a gap rather than an error, and a file that never states a training policy has not refused anything.
Testing a file properly
Reading a robots file is not the same as testing it, because the decision depends on the crawler and the path together. A path can be allowed for one agent and blocked for another, and the group logic above is exactly where a file stops doing what its author intended.
Enter a hostname and the tool fetches the file, lists the groups it found, and tests real paths against it. Each result says whether the path is allowed and which rule decided it, so a surprising answer points at the line responsible rather than leaving you to work it out.
Run it against the paths you care about and at least two user agents, one of them Googlebot. The case worth catching is a path that is allowed for everyone and blocked for Googlebot, or the reverse, because that is the shape of a file that was edited by somebody who assumed one group covered everything.
When you need Googlebot to crawl something you blocked
Blocking too much is a real failure, and it usually happens to stylesheets and scripts. Google renders pages to index them, and a renderer that cannot fetch your CSS and JavaScript sees a page that looks nothing like the one your visitors get.
If your pages are built by JavaScript, check that the paths serving those files are allowed. That single check explains a large share of pages that look fine in a browser and empty in search results. Once the file is right, the website audit reads twenty pages and reports what a crawler found on each one. and the sitemap guide covers the other half of the crawl problem, which is whether the URLs you are advertising can be read at all.
EnterraHost