To block crawling
Use robots.txt. This is what it is for, and it is the right tool for keeping a crawler out of paths it wastes time on, such as internal search results or faceted URLs.
Run a robots.txt check on any site. It reads the file, reports what is malformed, and tests real addresses against the rules the way a crawler applies them.
A robots.txt check tells you two things. Whether the file is readable and correctly formed, and whether the addresses you care about are actually allowed. A file can be valid and still block the pages you want crawled, which is why the file is only half the answer.
A content signal is a declaration rather
than an enforcement. Disallow says where a crawler may go. This says what
it may do with what it reads, and a crawler that ignores it is not blocked by it.
Matched in your browser against the rules above, so changing the path costs nothing.
Each verdict names the line that decided it, so a block is something you can fix rather than a red cross.
Reported, not scored. These matter only if you publish an agent API or an MCP server. A 404 here is not a fault for a content site.
Score: agent access 40, content signals 15, llms.txt and llms-full.txt 20, sitemap and Link headers 15, markdown negotiation 10.
The part people get wrong
Most people who search for "robots txt disallow" have one of three rules open
and want to know what it does. Disallow: / blocks the whole site. Disallow:
with nothing after it blocks nothing at all. Disallow: /private/ blocks one directory
and every path underneath it. A crawler matches from the start of the path, so the leading slash
matters and a rule without one is ignored.
None of those three removes a page from Google. A disallowed URL is not crawled, so it is never read, so a noindex on it is never seen - and a page linked from somewhere else can still be indexed on the strength of that link. Blocking and removing are different jobs: use robots.txt to stop a crawler wasting time, and use a noindex header or tag when the goal is to keep a page out of the index. Test the exact URL below with the crawler you care about - the answer depends on which group in the file applies to that agent, not on the file as a whole.
This is the single most expensive misunderstanding in the file. A URL you disallow is not removed from search, and can still rank. A crawler that is blocked never reads the page, so it never sees a noindex instruction, and a page linked from elsewhere can be indexed on the strength of that link alone.
Use robots.txt. This is what it is for, and it is the right tool for keeping a crawler out of paths it wastes time on, such as internal search results or faceted URLs.
Use a noindex robots meta tag or the equivalent HTTP header, and let the page be crawled so the instruction is actually read. Blocking the crawl and asking for noindex at the same time cancels the second request.
Matching
Disallow /private also blocks /private-notes and /private/report. A path with no wildcard and no end anchor matches anything that starts with it, which is why a rule that looks tidy can be wider than intended.
Allow and Disallow are not a first-match list. Every rule is tested and the most specific one, measured by pattern length, decides. A short Disallow cannot override a longer Allow.
When an Allow and a Disallow match with the same pattern length, the allow rule wins. That is the behavior in RFC 9309 and it is the opposite of what most people expect when they write the two lines in the wrong order.
Permissions
For most of the protocol's life, a robots file answered a single question about where a crawler may go. It had no vocabulary for what happens after the page is fetched, and that gap matters now that a reader can keep the text.
Content-Signal is a line that answers the second question. It carries one or more settings written as name and value pairs, and each is a separate decision. This tool reports every one it finds and explains what it permits.
Content-Signal: search=yes, ai-input=yes, ai-train=no
search=yes allows the page to be indexed and linked to.ai-input=yes allows the text to be read in order to answer a question, and quoted in the answer.ai-train=no withholds it from training.It is a declaration, not an enforcement. A crawler that ignores it is not blocked by it the way a Disallow line blocks one, so it states an intent rather than imposing one. A file that carries no Content-Signal is silent rather than permissive, and a reader that looks for one finds nothing to go on.
The tool also reports any Sitemap line and the address it declares, which is the same file the sitemap validator reads in full. Both are file level declarations rather than crawl rules, which is why neither is scoped to a user-agent.
Next
A correct robots file does not tell you whether the pages behind it resolve. Neither does the sitemap it names, unless that file is valid too. Check either one next.
The full walkthrough is in how a robots.txt rule decides, including why a blocked page can still appear in search.
Questions
Disallow: / with a single slash blocks
everything, while Disallow: with nothing after it blocks nothing. It does not remove
anything from Google: a blocked URL is never crawled, so a noindex behind it is never read, and a
page linked from elsewhere can still appear in results. To keep a page out of the index, use a
noindex header or meta tag, and let the crawler reach the page so it can see it.Content-Signal line carrying one or more name and value pairs. Three settings are in common use, search, ai-input and ai-train, and each is set to yes or no. Unlike a Disallow line it is not enforced by anything. A reader that chooses to honor it does, and one that does not is not stopped.no is a request rather than a block.