Anthropic's crawlers come in three flavors, and they do different jobs. If you are trying to work out whether Claude can find and cite your brand, the answer depends on which of the three you have allowed in robots.txt. Anthropic documents all of this publicly, so there is no guesswork required — but the three names are easy to confuse, and confusing them leads to the wrong fix.
The three Anthropic crawlers
Anthropic's documentation lists three user agents, and each one has a distinct purpose.
ClaudeBot collects content that may be used for model training. Blocking it excludes your future content from training. It has nothing to do with whether Claude retrieves your page when someone asks a question.
Claude-User fetches pages when a user asks Claude a question. If a person types "what does [your product] cost" into Claude and Claude goes to look, that fetch is Claude-User. Blocking it prevents retrieval for user queries — the closest thing to "Claude can't read my site when asked about it."
Claude-SearchBot indexes content to improve search results. Blocking it prevents indexing for search.
Anthropic says its bots honor robots.txt, support Crawl-delay, and publishes its IP ranges. That last point matters for verification: if you want to confirm that a request claiming to be ClaudeBot really came from Anthropic, you can check the source IP against the published ranges rather than trusting the user agent string.
What each block actually costs you
| User agent | What it does | If you block it |
|---|---|---|
| ClaudeBot | Collects content that may be used for model training | Future content is excluded from training |
| Claude-User | Fetches pages when a user asks Claude a question | Claude can't retrieve your page for user queries |
| Claude-SearchBot | Indexes content to improve search results | Your content isn't indexed for search |
A common mistake is blocking ClaudeBot for training reasons and assuming that also removes you from Claude's answers. It does not. If you want out of training but still want to be retrievable and citable, the specific block you want is ClaudeBot — leave Claude-User and Claude-SearchBot alone.
The reverse mistake is subtler. A site owner who wants maximum visibility sometimes blocks nothing and then wonders why a competitor gets cited. Training access and retrieval access are separate permissions, and only two of the three affect what a user sees.
How robots.txt actually resolves these rules
robots.txt follows RFC 9309, and two rules catch people out.
First, a crawler follows the most specific group that names it and ignores the * group in that case. So if you have a User-agent: * block with Disallow: / and a separate User-agent: ClaudeBot group that allows everything, ClaudeBot reads only its own group. The wildcard does not apply to it.
Second, within a group, the longest matching rule wins. Disallow: /pricing/ beats Allow: / for any URL under /pricing/, regardless of the order they appear in.
One more thing worth internalizing: robots.txt expresses intent. It is not access control. A crawler that ignores it — and Perplexity's user-triggered fetcher generally does, because the user requested the page — will still get through. Treat robots.txt as a signal to well-behaved crawlers, not a firewall.
A quick way to see where you stand
You can read your own robots.txt, but it is easy to misread a nested rule or miss a user agent you forgot you blocked. Our free AI crawler checker reads a site's robots.txt and reports, bot by bot, whether 16 crawlers such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot are allowed, partly restricted or blocked, and whether /llms.txt exists. It works from the file itself — it does not see server logs, traffic, visits or CDN and firewall blocks. So if the checker says ClaudeBot is allowed but your logs show nothing, the block is happening somewhere else: a WAF rule, a bot-management product, or a country-level filter.
What to check, in order
Start with the user agent you care about. If your goal is being cited in Claude's answers, the two that matter are Claude-User and Claude-SearchBot. ClaudeBot is a training decision and a separate conversation.
Then check for accidental blocks. A Disallow: / under User-agent: * will not stop ClaudeBot from reading your site if ClaudeBot has its own group, but it will stop every crawler that does not have a named group. Many sites have a wildcard block left over from a staging configuration.
Then check what you are actually serving. Being crawlable is necessary but not sufficient. Claude has to be able to read the page and find a clear answer. A pricing page that renders its numbers in client-side JavaScript, or a comparison page that never states what the product does, gives the crawler little to work with. Anthropic's documentation covers the crawling side; the retrieval and citation side depends on your content.
Finally, remember that changes take time to propagate. Anthropic does not publish a specific refresh window for robots.txt in the documentation cited here, so treat any timeline you see elsewhere with caution. If you unblock a crawler, give it days, not minutes, before concluding it did not work.
Where Claude fits in the wider picture
Claude is one of several assistants your buyers use, and each has its own crawler set and its own rules. Google's setup is different again — Google-Extended controls training and grounding for Gemini, while AI Overviews run through ordinary Googlebot crawling and indexing. If you are working across assistants, the same logic applies in each case: identify the crawler that governs the surface you care about, and check that specific permission.
Our write-ups on Google AI Overviews versus ChatGPT and on how grounded Gemini answers pick websites cover those two surfaces. For Claude specifically, the practical steps are the three above: know which agent does what, check for accidental wildcard blocks, and make sure the page you want cited says something worth citing.
Your next step
Run your domain through the AI crawler checker and read the ClaudeBot, Claude-User and Claude-SearchBot rows. If Claude-User or Claude-SearchBot shows as blocked and you did not intend that, fix the robots.txt group and note the date you changed it. If both are allowed, the constraint is not crawling — move on to the content itself.