Which AI crawlers your robots.txt allows, and how to say so

"Can AI read my site?" is really two questions, and they have different answers. One is whether an assistant may use your pages to answer a question and name you. The other is whether your pages may be used to train a model. Most businesses want the first and are indifferent, or hostile, to the second. robots.txt is where you say which.

It is a permission file, not a lock. A crawler that ignores it is not stopped by it, and a crawler that respects it will fetch nothing you disallow. Both of the big assistants publish that they honour it, which is why it is the right place to have the argument.

The agents, and what each one actually does

These are the user agents worth knowing. They belong to four companies and do three different jobs.

• GPTBot - OpenAI, collects pages for training. Blocking it does not stop ChatGPT answering questions about you.
• OAI-SearchBot - OpenAI, indexes pages so ChatGPT search can cite them. This is the one that decides whether you appear in an answer.
• ChatGPT-User - OpenAI, fetches a page when a person asks ChatGPT to look at it. Blocking it makes you invisible to that request.
• ClaudeBot - Anthropic, training.
• Claude-User - Anthropic, fetching on a person's behalf.
• PerplexityBot - Perplexity, indexing for citation.
• Google-Extended - Google, controls AI training and Gemini grounding. It has no effect on Google Search, which uses Googlebot.
• CCBot - Common Crawl, a public dataset many models train on. Blocking it does not affect any search engine.
• Bytespider - ByteDance.
• Applebot-Extended - Apple, AI training. Apple's search crawler is plain Applebot and is separate.
• Meta-ExternalAgent - Meta, training and product use.

If you only remember one line from this page: blocking the training crawler and allowing the search crawler is a coherent position, and it is the one most businesses end up at.

How the rules decide

robots.txt matches the most specific group. A crawler looks for a group whose User-agent line names it exactly; if there is none, it falls back to the group for *. That single rule explains almost every surprise:

• A Disallow: / under User-agent: * blocks every AI crawler you did not name, because they all fall back to it.
• Naming a bot in its own group overrides the wildcard for that bot - but only for that bot.
• Group names are case-insensitive; paths are not.
• A blank line inside a group does not end it. A new User-agent line does.

Three blocks you can copy

Allow everything. The default if you have no strong view. Nothing to add, but be explicit so the next person does not guess:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Allow citation, refuse training. The middle path. Search and answer engines may index and quote you; the training crawlers are told to stay out.

# Answer engines and search: welcome.
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-User
Allow: /

# Training crawlers: not welcome.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

# Everything else follows the ordinary rules.
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Block everything AI. A defensible choice for a site whose content is the product. Be aware what it costs: assistants will not be able to quote you, so they will describe you from other sources or not at all.

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

Sitemap: https://example.com/sitemap.xml

Say what they may do, not only where they may go

robots.txt can also carry a Content-Signal line, which states what a crawler may do with what it reads. It is a declaration rather than an enforcement, and it costs one line:

Content-Signal: search=yes, ai-input=yes, ai-train=no

That reads: appear in search, use as input to an answer, do not train on it. Change the last value if training is welcome. There is no penalty for having no opinion, but a site that says nothing looks the same as one that objects.

The trap worth checking for

The commonest mistake is a wildcard block with named exceptions that do not exist. This file looks like it allows GPTBot and does the opposite:

User-agent: *
Disallow: /

User-agent: GPTBot
Allow: /

It is correct, actually - the named group overrides the wildcard. The version that fails is the one where somebody thinks the wildcard is a default and adds the allowance to the wrong group, or spells GPTBot as GptBot in a path rule, or blocks by path (Disallow: /blog/) and wonders why an assistant quotes a page they moved.

Check it rather than assume it

Reading the file is not the same as knowing what it does. The robots.txt tester fetches the live file, evaluates each of these agents against it, and names the line that decided each verdict - which is the difference between "blocked" and "blocked by User-agent: *, so every unnamed crawler is affected too".

Fix the line it names, re-run it, and the verdict changes. That loop is the whole job.