What llms.txt actually is
llms.txt is a proposed file format, published at llmstxt.org, that gives language models a short, curated map of your site. You place a plain Markdown file at the root of your domain — https://example.com/llms.txt — and it lists the pages that matter most, with a one-line description of each.
The idea is simple. A model that wants to understand your site does not need your navigation menu, your cookie banner, or your 4,000 product pages. It needs your homepage, your pricing page, your docs, and a sentence explaining what each one is. llms.txt is a way to hand that over in a format a model can read in one pass.
That is the whole pitch. It is not a ranking mechanism, not a replacement for your sitemap, and not a standard that anyone is obliged to follow.
Where it came from
The proposal was published by Jeremy Howard and others in 2024 as a companion to the idea of serving clean Markdown versions of pages to LLMs. The format borrows from Markdown conventions: an H1 with your site name, a blockquote summary, optional prose, and then H2 sections containing lists of links.
The important word is proposal. There is no standards body behind it, no vendor commitment, and no crawler that is documented to require it.
Who actually reads it
This is where most articles about llms.txt get vague, so let's be direct.
As of the vendor documentation we can verify, none of the major AI companies publicly state that their crawlers read llms.txt. That includes OpenAI, Anthropic, Google, and Perplexity. Their documented crawlers — OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended — are described in terms of robots.txt behavior and content use, not llms.txt.
That does not mean nobody reads it. It means the reading is unofficial and unannounced. Some developer tools, IDE assistants, and third-party AI products have added support. Some agent frameworks look for it. But if you are waiting for confirmation that ChatGPT search consults your llms.txt before answering a question about your brand, that confirmation does not exist in public documentation.
Be skeptical of anyone who tells you otherwise. "Not publicly documented" is the honest answer, and it will stay the honest answer until a vendor says something different.
Why people still add one
The cost is close to zero. A single static file, no build step, no JavaScript, no server change. If there is any chance a tool reads it, and the downside is one file you can delete later, the expected value is positive for many sites.
There is also a second benefit that has nothing to do with crawlers: writing an llms.txt forces you to decide which five to twenty pages on your site actually explain what you do. That exercise is useful even if no model ever fetches the file.
A minimal example
The smallest useful llms.txt is a title, a summary, and a handful of links.
# Example Analytics
> Example Analytics is a web analytics tool for small ecommerce teams.
> It shows which products get viewed but not bought.
## Docs
- [Quickstart](https://example.com/docs/quickstart): Install the tracking script and see your first report.
- [Metrics reference](https://example.com/docs/metrics): Definitions for every number in the dashboard.
## Product
- [Pricing](https://example.com/pricing): Plans, seat limits, and what is included.
- [Integrations](https://example.com/integrations): Shopify, WooCommerce, and Stripe.
## Optional
- [Changelog](https://example.com/changelog): Release notes.
- [About](https://example.com/about): Who runs the company.
Note the structure. H1 is the site name. The blockquote is the summary a model can quote directly. H2 sections group links. Each link has a short description after a colon.
The Optional section is part of the proposal: it signals pages that can be skipped if a model is working under a context limit.
A fuller example
A larger site might add a short prose section, more granular groups, and explicit notes about what not to use.
# Example Analytics
> Example Analytics is a web analytics tool for small ecommerce teams.
> It shows which products get viewed but not bought, and why.
Example Analytics is used by roughly 2,000 stores. The company is based in
Austin and has been operating since 2019. The product is self-serve; there is
no sales call required to start.
## Docs
- [Quickstart](https://example.com/docs/quickstart): Install the tracking script in about ten minutes.
- [Metrics reference](https://example.com/docs/metrics): Definitions for every number in the dashboard.
- [Attribution model](https://example.com/docs/attribution): How last-touch and first-touch are calculated.
- [Data retention](https://example.com/docs/retention): How long events are stored on each plan.
## Product
- [Pricing](https://example.com/pricing): Three plans, billed monthly or annually.
- [Integrations](https://example.com/integrations): Shopify, WooCommerce, Stripe, and a REST API.
- [Security](https://example.com/security): Subprocessors, encryption, and SOC 2 status.
## Company
- [About](https://example.com/about): Team and company history.
- [Changelog](https://example.com/changelog): Release notes, newest first.
## Optional
- [Blog](https://example.com/blog): Occasional posts on ecommerce measurement.
- [Careers](https://example.com/careers): Open roles.
The prose paragraph after the blockquote is optional in the spec but useful in practice: it gives a model facts it can state without fetching another page.
What llms.txt does not do
It does not control crawling. That is robots.txt, and the two files do different jobs. robots.txt tells crawlers where they may go. llms.txt suggests what is worth reading. A crawler that ignores llms.txt will still index your site normally.
It does not override your robots.txt rules. If you block OAI-SearchBot, your pages are not shown in ChatGPT search answers regardless of what your llms.txt says. If you block ClaudeBot, future content is excluded from training. The llms.txt file sits on top of those decisions, not in place of them. If you are unsure which OpenAI crawler does what, our breakdown of GPTBot vs OAI-SearchBot vs ChatGPT-User covers the differences.
It does not guarantee anything. No file guarantees a mention in an AI answer. Models choose what to cite based on the question, the available sources, and their own training. llms.txt is an input, not a lever.
Should you add one
Add one if:
- Your site has a lot of pages and the important ones are hard to find.
- You already maintain a sitemap and want a human-readable companion.
- You want a five-minute experiment with no downside.
Skip it if:
- Your site is five pages and they are all linked from the homepage.
- You are hoping it will fix a visibility problem. It will not.
- You have not yet sorted out robots.txt. Do that first.
If you do add one, keep it current. A stale llms.txt that lists a pricing page you retired two years ago is worse than no file at all.
A practical next step
Open your sitemap, pick the ten pages that best explain what you sell and how it works, and write one sentence for each. If you want a starting template rather than a blank file, our free llms.txt generator builds the file from a site name, a summary, key facts, and your important links — everything stays in your browser, nothing is uploaded. Then publish it at your domain root and check it in a week to see whether your server logs show any requests.