A sitemap does not get you cited by ChatGPT. It tells crawlers which URLs exist and when they last changed, so they can spend limited crawl budget on pages that matter. For AI crawlers, that matters more than it used to, because several of them fetch pages on demand when a user asks a question and only index a subset of what they find.
What a sitemap actually does for AI crawlers
A sitemap is a list of URLs, optionally with a lastmod timestamp. It is a discovery aid, not a ranking factor and not a guarantee of inclusion. Google's documentation on sitemaps is explicit that submitting one is a hint, not a directive.
The same logic applies to AI crawlers. Each vendor runs different bots with different jobs:
- OpenAI's OAI-SearchBot surfaces sites in ChatGPT search. Sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers. GPTBot is separate and collects training content; blocking it does not affect search.
- Anthropic's Claude-SearchBot indexes content to improve search results, while ClaudeBot collects training content and Claude-User fetches pages when a user asks a question.
- Perplexity's PerplexityBot surfaces and links sites in Perplexity search results. Perplexity-User fetches pages for user questions and generally ignores robots.txt because the user requested the fetch.
- Google-Extended controls training and grounding for Gemini, not inclusion in Google Search. AI Overviews sit inside normal Search, so Googlebot crawling and indexing govern them.
None of these vendors publishes a documented preference for sitemaps over links. Treat the sitemap as one of several discovery paths, and keep the signals inside it honest.
Accurate lastmod, not today's date
The most common sitemap mistake is stamping every URL with the current date on every build. Crawlers that compare lastmod against what they already have learn quickly that your timestamps carry no information. At that point the field is ignored, and you have lost the one signal that could have prioritized your genuinely updated pages.
A defensible lastmod reflects a substantive change to the page: new copy, changed pricing, an updated comparison table, a revised spec. Cosmetic template changes, ad slot rotations, and re-rendered navigation do not count.
For example, suppose you publish a pricing page and update it monthly. If lastmod moves only when the numbers change, a crawler that revisits can trust the field. If it moves daily because your CMS touches every record on deploy, the field is noise.
Practical rules:
- Set
lastmodfrom the content's own modification timestamp, not the build timestamp. - Use W3C datetime format with a timezone, for example
2026-09-21T14:05:00-04:00. - Do not backdate. If you rewrote a page, say so with the real date.
- If you cannot produce a trustworthy timestamp for a URL, omit
lastmodfor that URL rather than guessing.
Segment your sitemaps
One giant sitemap mixing blog posts, product pages, help docs and tag archives gives crawlers no way to reason about freshness by section. Splitting by content type is cheap and makes the lastmod data more legible.
A workable split:
| Sitemap | Contents | Typical change rate |
|---|---|---|
| sitemap-products.xml | Product and pricing pages | Low, but high value |
| sitemap-docs.xml | Help and documentation | Moderate |
| sitemap-blog.xml | Articles and guides | High for new URLs |
| sitemap-legal.xml | Terms, privacy, policies | Very low |
Two things this buys you. First, a crawler that trusts your blog sitemap can spend its budget there instead of re-reading legal pages. Second, when something goes wrong in one section, the blast radius stays small.
Keep individual files under the size and URL limits the major search engines document, and keep a sitemap index if you have many files. If you want to signal which pages are worth reading in full, that is a different conversation — see llms.txt Explained: What It Is, Who Reads It, and Whether You Need One.
The robots.txt sitemap line
A sitemap can be discovered through Search Console, through a Sitemap: line in robots.txt, or by guessing the default path. The robots.txt line is the one you control without any vendor dashboard.
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap-index.xml
A few things to keep straight, because robots.txt rules are easy to misread:
- Per RFC 9309, a crawler follows the most specific group that names it. If a group names
OAI-SearchBot, that crawler ignores the*group entirely. - Within a group, the longest matching rule wins.
- robots.txt expresses intent. It is not access control. Perplexity-User generally ignores it because the fetch was user-initiated, and OpenAI notes that robots.txt rules may not apply to ChatGPT-User for the same reason.
- Changes are not instant. OpenAI and Perplexity both say robots.txt changes can take about 24 hours to take effect.
So the Sitemap: line is a pointer, and the allow/disallow rules around it decide whether a given bot reads anything at all. Get both right, or the sitemap is decoration.
IndexNow for Bing-powered experiences
IndexNow is a simple protocol: you send a URL to an endpoint, and participating search engines are notified that it changed. It is supported by Bing and several other engines, which means it can reach surfaces powered by Bing, including some AI experiences that draw on Bing's index.
What it is not: a way to notify OpenAI, Anthropic or Perplexity. None of those vendors documents IndexNow support. If you see a tool claiming otherwise, ask for the vendor documentation.
Where IndexNow is worth the effort: sites that publish or update URLs frequently and want faster pickup in Bing-backed results. The implementation is a single POST with a key file hosted on your domain. For a site with a few dozen pages, it is optional. For a site publishing daily, it is a reasonable addition to the sitemap workflow.
What to check this week
Pull your sitemap index and open the file with the highest change rate. Look at the lastmod values. If they all match the date of your last deploy, you have a fixable problem: wire the field to real content modification times, then split the file by section so crawlers can tell your fresh pages from your static ones. If your pages rely on client-side rendering to show that content at all, read Can AI Crawlers Read Your JavaScript? Why Server-Rendered HTML Still Wins before you touch the sitemap. Once those two things are clean, a Sitemap: line in robots.txt and an IndexNow ping on publish are the last ten minutes of work.
If you want to know whether any of this is landing, PromptRanking tracks whether ChatGPT, Gemini and Perplexity mention a brand in their answers, which is a more direct read than crawl logs.