Most teams find out the hard way: a page looks perfect in a browser, but the AI answer that cites competitors never mentions them. One common cause is that the facts on that page only exist after JavaScript runs, and the fetcher that arrived never ran it.
What the vendors actually document
Start with what is confirmed, because the gap between what crawlers do and what people assume they do is where most confusion lives.
OpenAI. OAI-SearchBot surfaces websites in ChatGPT search, and sites that opt out of it are not shown in ChatGPT search answers. GPTBot collects content to improve foundation models, and blocking it does not affect ChatGPT search. ChatGPT-User handles user-initiated actions in ChatGPT and custom GPTs; because a user starts those actions, robots.txt rules may not apply, and it is not used to decide whether content appears in search.
Anthropic. ClaudeBot collects content that may be used for training. Claude-User fetches pages when a user asks Claude a question, and blocking it prevents retrieval for those queries. Claude-SearchBot indexes content to improve search results, and blocking it prevents indexing for search.
Perplexity. PerplexityBot surfaces and links sites in Perplexity search results and is not used to crawl content for foundation models. Perplexity-User fetches pages for user questions and generally ignores robots.txt because the user requested the fetch.
Google. Google-Extended controls training and grounding for Gemini, not inclusion in Google Search. AI Overviews are part of Search and follow normal Googlebot crawling and indexing.
Notice what these descriptions do not say. None of them state whether the crawler executes JavaScript, how long it waits for a page to render, or whether it renders at all. Those details are not publicly documented. Anyone who tells you "PerplexityBot renders JavaScript" or "ClaudeBot doesn't" is going beyond what the vendors have published.
That uncertainty is the argument for server-rendered HTML. If you do not know whether the fetcher runs your JavaScript, do not make your content depend on it.
Why client-side rendering is a gamble
With client-side rendering, the HTML the server returns is mostly a shell: a script tag, a root div, and little else. The headline, the pricing table, the product specs, the FAQ answers — all of it gets injected after the framework boots and the data arrives.
A browser does that work in a fraction of a second on a fast connection. A fetcher deciding whether to cite you may not. It may pull the raw response, extract text, and move on. In that case your page looks, to the crawler, like an empty container.
This is the same problem that made client-side rendering risky for traditional search, and it is more acute here. Search engines invested heavily in rendering pipelines over many years. AI fetchers are newer and less predictable. Some are optimized for speed and volume, not for running a React app.
There is also a cost angle. Rendering every page is expensive. A crawler that wants to cover millions of pages has a reason to prefer cheap HTML fetches, and nothing in the public documentation promises otherwise.
Test what a crawler sees, in two minutes
The fastest check does not require any crawler at all. Fetch the raw HTML the way a simple client would, using a command-line tool that does not run scripts.
Or open the page in a browser, right-click, and choose View Page Source. That shows the initial response, not the DOM after scripts run.
Now ask a blunt question: are your key facts in that output?
- The product or service name, in text
- The specific claim you want quoted (pricing, coverage, compatibility, limits)
- Names of the people, places, or entities involved
- Headings that describe what the page is about
- Any structured data you rely on
If the answer is no for most of these, you have found the problem. If the answer is yes, you have removed one variable and can look elsewhere.
For a fuller picture, compare the raw response against the rendered DOM in your browser's inspector. The difference between them is exactly what a non-rendering fetcher would miss.
Server-rendered HTML as the default
Server-side rendering (SSR) and static site generation (SSG) both put content in the initial response. The server builds the HTML with the text already in it, and JavaScript enhances the page afterward rather than producing it.
For most marketing sites, docs, and content pages, that is the right default. You keep the interactive parts — search boxes, filters, calculators — and you stop hiding the substance behind them.
A practical pattern: render the page's core content on the server, then hydrate. Users get the same experience. Crawlers get the text immediately.
The same logic applies to structured data. If you mark up a product, an article, or an FAQ, that markup should be in the server response, not injected later. For guidance on which types are worth the effort, see Schema Markup for AI Search: Which Types Are Worth Your Time.
What to move into the HTML
Not everything needs to be server-rendered. Prioritize what you want quoted.
| Keep in server HTML | Can stay client-side |
|---|---|
| Headlines and subheads | Interactive filters and sort controls |
| Product and service descriptions | Personalized recommendations |
| Pricing and plan details | Live inventory widgets |
| FAQ answers | Chat widgets |
| Author, company, and contact details | Animation and hover states |
| Article body text | Session-specific content |
A simple rule: if you would want an AI answer to repeat it, it belongs in the HTML.
Rendering is one layer, not the whole story
Getting your content into the response is necessary but not sufficient. Crawlers also need permission and a reason to come back.
Check that the bots you care about are not blocked. For OpenAI specifically, the distinction between the three user agents matters, and the choices are not interchangeable — GPTBot vs OAI-SearchBot vs ChatGPT-User: Which OpenAI Crawler Should You Allow? walks through it. Remember that robots.txt changes can take about 24 hours to take effect for OpenAI and Perplexity, and that robots.txt expresses intent rather than acting as access control.
If you are weighing whether to publish an llms.txt file, that is a separate question from rendering, and the honest answer is that adoption is uneven — llms.txt Explained: What It Is, Who Reads It, and Whether You Need One covers what is known.
When you genuinely need client-side rendering
Sometimes you do. Logged-in dashboards, heavy applications, and highly personalized views are not going to be static.
In those cases, consider a server-rendered surface for the parts you want discovered. A public marketing page, a docs page, or a pricing page can be static even if the app behind it is not. You do not have to choose one architecture for the whole site.
If you cannot avoid client-side rendering for a key page, at minimum put a clear text summary in the initial HTML — a short paragraph that states what the page offers and who it is for. It is not a substitute for full content, but it gives a non-rendering fetcher something to work with.
Check one page today
Pick the page you most want cited in AI answers. Fetch the raw HTML and read it the way a simple client would. If your key facts are missing from that response, that is your next fix — and it is usually a smaller change than it sounds.