Bot-only sitemap explained

A bot-only sitemap is the sitemap served to AI agents, separate from the one Google and Bing read. It lists the pages you want AI agents to read, including agent-only pages that have no human counterpart, and leaves out the pages you do not. Search engines keep receiving your existing sitemap unchanged.

The idea sounds small. It is one of the few levers you have over what AI agents read first, because indexers and retrieval bots check the map before they read anything else.

Why agents should not read Google's sitemap

Your sitemap.xml was built for one reader with one job: index every page so it can be ranked. It is usually generated by the CMS, covers everything, and nobody has pruned it since launch. Three things go wrong when AI agents read it.

Crawl budget goes to the wrong pages. An AI search indexer sweeps a sitemap on a schedule and fetches what it finds. If 60% of the URLs are tag archives, retired products and pagination, 60% of the indexer's attention on your site is spent there, and the pages that answer buyer questions wait.

Expired URLs become answers. A discontinued product that is still in the sitemap is still a page an assistant can read and recommend. Stale and hallucinated links are already measured in AI answers (OpenAI's referral 404 rate rose from 0.3% to 3.7% between Q1 and Q2 2025). A sitemap that lists dead pages contributes.

Agent-only pages have nowhere to live. A comparison page, a spec sheet as text or a safety FAQ written for agents does not belong in the human sitemap, because it has no human page behind it. It needs a map of its own.

How indexers and retrieval bots use the map

Two classes of agent read the sitemap, differently.

AI search indexers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer) do shallow, periodic sweeps. They fetch robots.txt, follow it to the sitemap, and work through the URLs to keep an index fresh. On tollbit.com over the past month, OAI-SearchBot's single most-fetched URL was robots.txt, about 1,000 times, ahead of the homepage. Claude-User's most-fetched URL was also robots.txt, nearly 600 times. For these agents the sitemap is the front door.

Retrieval bots (ChatGPT-User, Claude-User, Perplexity-User) fetch a specific page because a user asked a specific question. They arrive from a search result or from a link on a page they already fetched. They use the sitemap less directly, but the index they searched was built from it, and the links they follow come from pages that were in it.

So the bot-only sitemap does two jobs: it shapes what indexers hold about you, and it seeds the link graph retrieval bots walk.

What goes in it

What stays out: pagination, tag and category archives, retired products, campaign landing pages that have ended, legal boilerplate, and anything the human sitemap carries only because the CMS generated it.

Per-platform maps

Agents differ. One platform's indexer sweeps often and reads everything, another's fetches only what users ask about. A bot-only sitemap can be served per platform, so the list OAI-SearchBot receives can differ from the one PerplexityBot receives, and the robots.txt each reads can point to its own map. This is the same reason you serve per-platform page versions: the facts are constant, the delivery suits the reader. Recrawl intervals vary by platform, from hours to weeks, so a page added to the map today is in some indexes tomorrow and others next month.

A short example

The robots.txt served to bots points at the bot-only map:

User-agent: *
Sitemap: https://yourbrand.com/sitemap-agents.xml

The human robots.txt, served to Googlebot and Bingbot, is unchanged and points at the sitemap it always has.

Where TollBit fits

TollBit serves the robots.txt and sitemap that identified AI agents read, because it is the origin for bot traffic on your domain. Agent-only pages published in the Agent Site CMS are added to the bot-only sitemap on publish, and server logs show which agents fetched the map and which pages they read afterwards.

Frequently asked questions

Does a bot-only sitemap change what Google sees? No. Googlebot and Bingbot are never routed to the agent site. They receive your existing robots.txt and sitemap exactly as before.

Is a separate sitemap for bots the same as llms.txt? No. llms.txt is a hint file that an agent may or may not read. A sitemap is the standard discovery mechanism every indexer already uses, and the bot-only version is that mechanism with a different list of URLs.

How quickly do agents pick up a new page in the map? It varies by platform, from hours to weeks. Indexers that sweep frequently see it first. Retrieval bots see it when a user's question leads there, which is why linking the new page from pages agents already read matters as much as the sitemap entry.

Related