llms.txt vs an agent site: do you need both?
llms.txt is a Markdown file at the root of your domain that lists the pages you would like a language model to read. An agent site is the version of your website that AI agents are actually served when they fetch any URL. The first is a suggestion that nobody is obliged to follow. The second is the response itself. If you can only do one, change the response.
What llms.txt is
The llms.txt proposal was published in September 2024 by Jeremy Howard of Answer.AI. It asks site owners to place a file at /llms.txt containing a title, a short summary and sections of links with one-line descriptions, so that a model with a small context window can find the pages that matter. An optional /llms-full.txt holds the full text of those pages in one file.
It is cheap to produce, and several documentation hosts now generate it automatically. For a developer product with a docs site, it is a tidy table of contents.
Where llms.txt stops
- Nobody has to read it. As of this writing no major AI platform has published a commitment to fetch llms.txt before answering, the way search engines committed to robots.txt and sitemaps. In our logs across 10,000+ sites, retrieval bots overwhelmingly fetch ordinary HTML pages, and alternative bot-facing files see marginal volume.
- It does not change the page. If an agent follows a link from llms.txt and lands on a 900 KB page with a cookie wall and a JavaScript shell, the hint file has not helped. The fetch still returns what your human site returns.
- It is a single list. One file for every agent, with no way to serve a retrieval bot, a search indexer and a computer-use agent differently, and no measurement of who read it.
- It can drift. A hand-maintained list falls out of date the week a product page moves.
What an agent site does
An agent site is the agent-facing version of your website on your own domain. A CDN or WAF rule routes identified AI agents to a tollbit.yourbrand.com origin. Humans and Googlebot are never routed. From that origin:
- Every fetch returns server-rendered text with the same facts as the human page, without overlays and scripts. A 1,201 KB article becomes 41 KB.
- The robots.txt and sitemap that bots read are served from the agent site, so new agent-only pages (comparisons, spec sheets, FAQs) are discoverable without changing the human sitemap.
- Pages can differ per platform, and a human clicking a cited agent-only page lands on the nearest real page, never a 404.
- Reads are measured in the same server logs, per agent and per page. Bots do not run JavaScript, so this is the only place they show up.
Side by side
| llms.txt | Agent site | |
|---|---|---|
| What it is | A Markdown index at your root | The origin AI agents are served from |
| Who reads it | Agents that choose to look for it | Every identified AI agent, on every fetch |
| Changes the page an agent gets | No | Yes |
| Per-platform versions | No | Yes |
| Discovery of new pages | Only via the list | Bot-served robots.txt and sitemap |
| Measurement | None built in | Reads per agent, per page, from logs |
When llms.txt is worth adding anyway
Add it if you run documentation or a developer product. Coding agents are the one class of visitor that is sometimes instructed to look for it, and on tollbit.com the claude-code user agent fetched our State of the Bots report about 1,350 times in 30 days, more than any other page. Developers are piping reports into their agents, and a table of contents helps them.
Add it if it costs you nothing. Serve it from the agent site so it is generated from the pages you have actually published to bots, rather than maintained by hand. Then it stays true, and you can see in the logs whether anyone fetched it.
Frequently asked questions
Do ChatGPT, Claude or Perplexity read llms.txt? None of them has published documentation saying their retrieval bots fetch it. Our logs show robots.txt among the most fetched URLs by several major agents. llms.txt does not appear in that list.
Is llms.txt the same thing as robots.txt for AI? No. robots.txt tells crawlers what not to fetch and is honoured unevenly. llms.txt suggests what to read and carries no rule at all. Neither changes the content of a page.
Can an agent site generate my llms.txt? Yes. Because the agent site already holds the bot-facing version of each page and the bot-only sitemap, the index file can be produced from them and kept current automatically.