Training bots vs RAG bots vs AI search indexers: what is the difference?
They differ by what happens to the page after the fetch. A training bot copies it into a dataset that shapes a future model. A retrieval (RAG) bot fetches it live, because a person just asked a question, and the page becomes part of the answer. An AI search indexer stores it in the index the assistant searches before fetching anything live. Each major AI company runs one of each, under a different user agent, and each class deserves a different decision from a site owner.
The three classes, with their user agents
| Class | User agents | Trigger | What the page becomes |
|---|---|---|---|
| Training bots | GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Bytespider (ByteDance), meta-externalagent (Meta), Google-Extended (Google) | Operator's crawl schedule | Training data for a model |
| Retrieval (RAG) bots | ChatGPT-User (OpenAI), Claude-User (Anthropic), Perplexity-User (Perplexity), Amzn-User (Amazon), MistralAI-User, Google-Agent | A person's question or task, right now | Context for one answer, then discarded |
| AI search indexers | OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), PerplexityBot (Perplexity), meta-webindexer (Meta), Amazonbot (Amazon) | Periodic sweep of robots.txt and sitemap | An entry in the index the assistant searches |
What each class looks like in server logs
On tollbit.com, Sep 15 to Oct 14 2026:
- Indexers sweep. meta-webindexer made 16.3K successful reads, more than any other agent, moving across the site from the sitemap. OAI-SearchBot (1.4K) fetched robots.txt about 1,000 times, more than any page. PerplexityBot made 1.9K reads while Perplexity-User, the retrieval side, made 0.4K. Index-heavy, retrieval-light.
- Retrieval bots read like a person. ChatGPT-User (10.5K reads) went to the homepage, publisher solutions, licensed RAG and the bot paywall page, and to one explainer post. Claude-User (3.2K) fetched robots.txt nearly 600 times first, then the State of the Bots report. These fetches track what people ask, not a crawl schedule.
- Training bots take what is allowed. ClaudeBot made 1.8K reads, CCBot 1.2K, Bytespider 1.0K, GPTBot 0.7K, meta-externalagent 2.3K. Steady, broad, and the class most likely to honour a robots.txt disallow.
A fourth visitor fits none of the three. The coding agent user agent claude-code fetched our State of the Bots report about 1,350 times, more than any other page. That is a developer's agent reading a report on instruction, the leading edge of computer-use agents.
What each class means for a site owner
Training bots are the one-time sale. Robots.txt is mostly honoured here, so this is the class a disallow actually affects.
Retrieval bots are the recurring visitor. They cannot cache the internet, because the next question needs the current page, so they come back. They are the class that puts your facts in front of a buyer at the moment of decision, and the class that often treats robots.txt as not applying to a user's request. Blocking them removes you from answers.
AI search indexers decide whether a retrieval bot ever finds you. They read robots.txt and the sitemap, so they are the class a bot-only sitemap speaks to, and they generally honour robots.txt. An indexer that cannot read your site means no live fetch later.
Allow, block or charge
| Class | Default for most sites | Why |
|---|---|---|
| AI search indexers | Allow, slim format | They send the retrieval bots and the clicks |
| Retrieval (RAG) bots | Charge per fetch, or allow | Recurring, high intent, carries your facts into answers |
| Training bots | Charge at a higher rate, or refuse | One-off copy, no presence benefit, robots.txt mostly works |
A publisher usually charges retrieval and prices or refuses training. A brand usually allows everything that produces answers. Both need the same identification layer, because 12.7% of traffic claiming to be Claude-User in H1 2026 was spoofed.
Where TollBit fits
TollBit identifies all three classes at the edge of 10,000+ sites, applies a per-class rate card through the bot paywall, and reports reads by agent and purpose in server-side analytics. The Bot Index lists each agent, its operator, its class and its published verification details.
Frequently asked questions
Can robots.txt tell the three classes apart? Yes, by user agent token. Disallow GPTBot while allowing OAI-SearchBot and OpenAI will honour both. Retrieval bots such as ChatGPT-User often operate outside robots.txt, so for that class the rule is advisory.
Is Amazonbot a training bot or an indexer? In behaviour it sweeps like an indexer, and Amazon documents it as powering Alexa and other services. It made 2.9K reads on tollbit.com in 30 days, while Amzn-User, the retrieval agent, appeared in single digits. Treat Amazonbot as an indexer with shopping intent behind it.
Should a brand block any of the three? Rarely. A brand wants its facts in answers, which means allowing indexers and retrieval bots. Training bots are a judgement call about future models, and the one class where a disallow reliably works.