We requested the robots.txt file from 210 well-known websites on 2026-09-21, using a declared bot user agent, and evaluated the rules with an RFC 9309 parser. The goal was simple: find out how many big sites are blocking AI crawlers, which crawlers they block, and whether they treat training bots differently from the bots that fetch pages for live AI answers.
Of the 210 sites, 166 returned a readable robots.txt. The other 44 refused the request or timed out for our declared bot — edge bot protection, not a robots.txt rule. Those 44 are excluded from every percentage below. That exclusion is worth remembering: the real share of sites restricting AI crawlers is probably higher than what we measured, because some of the sites we could not read are blocking before robots.txt is even served.
The headline numbers
Among the 166 readable files:
- 33.7% block at least one AI training crawler.
- 19.9% block at least one AI search or live-fetch crawler (OpenAI, Anthropic, Perplexity search or user agents).
- 14.5% block training crawlers but leave AI search crawlers open.
That third number is the interesting one. Roughly one in seven sites has made a deliberate distinction: don't train on us, but do feel free to cite us. It is a coherent position, and it lines up with how the vendors describe their own bots.
Meanwhile, Googlebot and Bingbot were blocked site-wide by almost no one — under 1% in both cases.
Training bots vs. search bots
The distinction that matters here is not "AI" versus "not AI." It is what the crawler is doing.
Training crawlers collect content that may be used to improve or train foundation models. GPTBot is OpenAI's example; ClaudeBot is Anthropic's; Google-Extended is Google's control for future Gemini training and Gemini grounding; Applebot-Extended, Meta-ExternalAgent, Amazonbot, CCBot and Bytespider sit in the same category. Blocking these does not remove you from the corresponding search product, because they are not the crawlers that decide whether your page appears in an answer.
Search and live-fetch crawlers are the ones that surface or retrieve pages when someone asks a question. OAI-SearchBot surfaces websites in ChatGPT search, and sites opted out of it are not shown in ChatGPT search answers. PerplexityBot surfaces and links sites in Perplexity results. Claude-SearchBot indexes content to improve search results. Claude-User and Perplexity-User fetch a page because a specific user asked about it.
That last group behaves differently from ordinary crawlers. Anthropic says its bots honor robots.txt, and blocking Claude-User prevents retrieval for user queries. Perplexity says Perplexity-User generally ignores robots.txt because the user requested the fetch. OpenAI makes a similar point about ChatGPT-User: because the action is user-initiated, robots.txt rules may not apply, and it is not used to decide whether content appears in search.
So a robots.txt block on a user-agent is a statement of intent. It is not access control, and for user-triggered fetches it may not be honored at all.
Who gets blocked, bot by bot
Share of the 166 readable sites blocking each crawler site-wide:
| Crawler | Type | Blocked by |
|---|---|---|
| CCBot | Training | 27.1% |
| Bytespider | Training | 24.7% |
| ClaudeBot | Training | 24.1% |
| Meta-ExternalAgent | Training | 22.3% |
| GPTBot | Training | 19.9% |
| PerplexityBot | Search | 19.3% |
| Google-Extended | Training | 19.3% |
| Applebot-Extended | Training | 18.7% |
| Amazonbot | Training | 17.5% |
| Claude-User | Live fetch | 16.3% |
| Perplexity-User | Live fetch | 15.7% |
| Claude-SearchBot | Search | 15.7% |
| ChatGPT-User | Live fetch | 12.7% |
| OAI-SearchBot | Search | 10.8% |
| Googlebot | Search | 0.6% |
| Bingbot | Search | 0.6% |
The pattern is consistent: the more a bot looks like a pure training collector, the more often it is blocked. The more it looks like a normal search crawler, the more it is tolerated. OAI-SearchBot, at 10.8%, is blocked roughly half as often as GPTBot at 19.9% — even though both belong to OpenAI. Sites are reading the documentation and acting on it.
One oddity worth flagging: ChatGPT-User (12.7%) is blocked more often than OAI-SearchBot (10.8%), despite both being tied to ChatGPT. Some of that is probably blanket rules written before the individual user agents were well understood.
Industry differences are enormous
| Industry | n | Block a training bot | Block a search bot |
|---|---|---|---|
| News | 24 | 92% | 79% |
| Reference | 15 | 73% | 33% |
| Media | 12 | 50% | 42% |
| Local | 9 | 44% | 22% |
| Travel | 12 | 25% | 0% |
| SaaS | 48 | 13% | 0% |
| Ecommerce | 24 | 13% | 8% |
| Education | 8 | 13% | 0% |
| Finance | 14 | 0% | 0% |
News is in a category of its own: 92% block at least one training crawler and 79% block at least one search crawler. That is close to a blanket refusal, and it makes sense for publishers whose business model depends on people visiting the page rather than reading a summary.
Finance is the mirror image. None of the 14 finance sites in our sample blocked a training crawler or a search crawler. That is a small sample, and it may reflect a preference for being present and correctly attributed in AI answers over restricting access.
SaaS and travel sit at the permissive end too — 13% and 25% on training, 0% on search. Ecommerce blocks search crawlers slightly more often (8%) than SaaS, travel or education, which is plausible given product data and price scraping concerns.
Note the sample sizes. Education (n=8) and local (n=9) are small enough that a single site moves the percentage by more than ten points. Treat those rows as directional, not precise.
Why 14.5% block training but keep search open
The logic is straightforward once you separate the two functions. A training crawler takes your content and turns it into model weights. You get nothing back — no visit, no citation, no link. A search or live-fetch crawler retrieves your page to answer a specific question, and in the documented behavior of OAI-SearchBot and PerplexityBot, it surfaces and links the source.
So a site can decide: no free training data, but yes to being cited. That is the position 14.5% of our readable sample has taken, and it is the most defensible middle ground we saw in the data.
The catch is that this position depends on the vendor honoring the distinction. OpenAI documents that blocking GPTBot does not affect ChatGPT search, and that OAI-SearchBot is what governs search inclusion. Anthropic separates ClaudeBot from Claude-SearchBot in the same way. Where the documentation is clear, the split works. Where it is not — user-triggered fetches, for instance — a robots.txt block may simply not be applied.
What this study does not tell you
Four limits, stated plainly:
- The sample is hand-picked, not random. We chose 210 well-known sites across nine industries. It is not a representative sample of the web, and the industry percentages should be read as snapshots of those specific lists.
- robots.txt shows intent, not enforcement. A rule means the site asked crawlers not to fetch. It does not prove the crawler complied, and it does not prove the site would notice if one did not.
- CDN and firewall blocks are invisible here. A site can block AI crawlers at the edge without writing anything in robots.txt. Our 44 unreadable files are evidence that this happens.
- robots.txt has its own parsing traps. Per RFC 9309, a crawler follows the most specific group that names it and ignores the
*group in that case. Within a group, the longest matching rule wins. A file can look like it blocks GPTBot while a more specific rule allows it, or the reverse.
Check your own file
If you want to know where you stand, our free AI crawler checker reads a site's robots.txt and reports, bot by bot, whether 16 crawlers such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot are allowed, partly restricted or blocked, and whether /llms.txt exists. It reads the file only — it does not see server logs, traffic, visits or CDN and firewall blocks, so a site can look open in the checker and still be blocked at the edge.
Run it on your own domain first. Then run it on two competitors in your industry and compare. If you find a training bot blocked and a search bot allowed, that is the 14.5% position, and it is worth confirming it is intentional rather than inherited from a template nobody has edited since 2024.