OpenAI's crawlers do different jobs, and blocking one does not necessarily block the others. The three names you'll see most often in server logs and robots.txt debates are GPTBot, OAI-SearchBot and ChatGPT-User. This piece explains what each one does according to OpenAI's documentation, shows robots.txt snippets for three common policies, and lays out what you give up with each.
What each agent actually does
GPTBot
GPTBot collects content that may be used to improve OpenAI's foundation models. Blocking it excludes future content from training. That is the entire scope of the decision: training data, not search visibility.
OAI-SearchBot
OAI-SearchBot surfaces websites in ChatGPT search. OpenAI's documentation is direct about the consequence: sites opted out of OAI-SearchBot are not shown in ChatGPT search answers. If you want to appear in ChatGPT's search results, this is the bot that decides it.
ChatGPT-User
ChatGPT-User is used for user-initiated actions in ChatGPT and custom GPTs. When someone asks ChatGPT to open a page or take an action on a site, this agent makes the request. Because the request is initiated by a user, robots.txt rules may not apply to it. OpenAI also states it is not used to decide whether content appears in search.
That last point matters more than it first appears. ChatGPT-User is a fetching agent, not an indexing agent. Blocking it does not remove you from ChatGPT search, and allowing it does not add you.
The short version
| Agent | Purpose | Affects ChatGPT search visibility? |
|---|---|---|
| GPTBot | Collects content for foundation model training | No |
| OAI-SearchBot | Surfaces sites in ChatGPT search | Yes |
| ChatGPT-User | Fetches pages for user-initiated actions | No |
One operational note from OpenAI: robots.txt changes can take about 24 hours to take effect.
A robots.txt detail that trips people up
Before the snippets, one rule from RFC 9309 that changes how these directives behave. A crawler follows the most specific group that names it and ignores the * group in that case. Within a group, the longest matching rule wins.
This means a robots.txt like this does not do what many people assume:
User-agent: *
Disallow: /
User-agent: OAI-SearchBot
Allow: /
OAI-SearchBot matches its own named group, so it ignores the * group entirely and follows Allow: /. If you intended to block everything except OAI-SearchBot, this file achieves it. If you intended something else, it probably doesn't. Write explicit groups for each agent you care about, and don't rely on * to fill gaps.
Also worth stating plainly: robots.txt expresses intent. It is not access control. A crawler that ignores it is not blocked by it.
Policy 1: Allow all three
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
Or simply omit the directives, since the default is to allow. This is the maximum-visibility position: training inclusion, ChatGPT search inclusion, and no friction for user-initiated fetches.
Trade-off: you accept that your content may be used to improve OpenAI's models, and you have no lever if that changes your mind later beyond editing robots.txt and waiting roughly a day.
Policy 2: Block training, keep search
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
This is the most common split policy, and it maps cleanly onto the distinction OpenAI draws. You keep your pages eligible to surface in ChatGPT search while opting out of foundation-model training.
Trade-off: none for search visibility, per OpenAI's documentation. The cost is whatever value you assign to being in training data, which is a judgment call about the future rather than a measurable present-day return. Note that blocking GPTBot does not affect ChatGPT search, so this policy is internally consistent.
Policy 3: Block all three
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
Trade-off: you are opting out of ChatGPT search results. OpenAI states that sites opted out of OAI-SearchBot are not shown in ChatGPT search answers. That is a real visibility loss if your buyers ask ChatGPT questions in your category.
The ChatGPT-User line is the murkier one. Because those fetches are user-initiated, robots.txt rules may not apply, so this directive may not have the effect you expect. If your actual concern is a user asking ChatGPT to read your page, robots.txt is the wrong instrument. Server-side controls are the right conversation, and that is outside what robots.txt can express.
How to check what you're actually serving
Most sites do not have one clean robots.txt. They have a CMS default, a plugin's additions, a CDN rule, and something a contractor added in 2023. Reading the file tells you what you're publishing; it does not tell you what is reaching you.
Our free AI crawler checker reads a site's robots.txt and reports, bot by bot, whether 16 crawlers such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot are allowed, partly restricted or blocked, and whether /llms.txt exists. It is a fast way to confirm that the policy you intended is the policy in the file. It does not see server logs, traffic, visits or CDN and firewall blocks, so treat it as a check on your published rules, not on crawler behavior.
That distinction matters because the two failure modes look identical from the outside. A crawler can be allowed in robots.txt and still never reach you because of a WAF rule. Or it can be blocked in robots.txt and still hit your origin because a CDN is caching or a rule is misconfigured.
Context: the other vendors draw the same lines
The training-versus-retrieval split is not unique to OpenAI. Anthropic separates ClaudeBot (training), Claude-User (fetching pages when a user asks Claude a question) and Claude-SearchBot (indexing for search). Perplexity separates PerplexityBot (surfacing and linking sites in search results, not training) from Perplexity-User (fetching pages for user questions, generally ignoring robots.txt because the user requested it).
Google's split is different in kind. Google-Extended controls whether content is used to train future Gemini models and for grounding in Gemini apps and the Vertex AI Gemini API. It does not affect inclusion in Google Search and is not a ranking signal. AI Overviews are part of Google Search, so they are governed by normal Googlebot crawling and indexing, not by Google-Extended.
The pattern across vendors: training controls and search controls are separate switches, and the names are similar enough to be confused in a config file.
Pick a policy, then verify the file
Decide which of the three positions above you actually want, write explicit User-agent groups for GPTBot, OAI-SearchBot and ChatGPT-User rather than leaning on *, and remember the roughly 24-hour propagation window before you judge the result. Then run your domain through the AI crawler checker and confirm the file says what you meant. If it doesn't, you've found the problem before it cost you anything.