What are ClaudeBot, Claude-User and Claude-SearchBot?

Anthropic runs three separate agents on the web: ClaudeBot collects pages for model training, Claude-User fetches a page live when a person asks Claude a question, and Claude-SearchBot builds the index Claude's web search draws on. They have different jobs, different volumes and different value to a site owner, so treat them as three visitors, not one.

The three agents side by side

Agent	Job	robots.txt token	Reads on tollbit.com, Sep 15 to Oct 14 2026
ClaudeBot	Training crawl	`ClaudeBot`	1.8K
Claude-User	Live retrieval for a user's question	`Claude-User`	3.2K
Claude-SearchBot	Index for Claude's web search	`Claude-SearchBot`	0.4K

Each identifies itself with its token in the user-agent string plus an anthropic.com contact address, and Anthropic documents all three on its support site with a robots.txt token for each. Our logs show Claude-User fetching robots.txt before it fetches content pages, which is what a fetcher that reads the file looks like.

What each one did on our site

Claude-User is the busiest Anthropic agent on tollbit.com and one of the most robots.txt-conscious bots we see. Its single most-fetched URL is /robots.txt, nearly 600 fetches in 30 days, more than any content page. Second is the State of the Bots report. That is a check-then-fetch pattern: confirm what may be read, then read it. Because it is retrieval traffic, its volume follows human questions. It rises when people ask Claude about AI bots and falls when they stop.

ClaudeBot's 1.8K reads have the shape of a crawl: breadth over depth, pages touched in sweeps, no clustering around the pages people are asking about that week. That is what training traffic looks like, and it is the one-off kind of value. A page crawled for training is worth something once.

Claude-SearchBot's 0.4K is small, and small is normal for an indexer on a site with a few hundred pages. Indexers sweep periodically to keep an index fresh. They do not fetch per question.

For scale, ChatGPT-User fetched 10.5K pages on our site in the same window, about three times Claude-User, and the three Anthropic agents together came to about 5.4K reads.

The spoofing problem

In H1 2026, 12.7% of traffic claiming to be Claude-User across the TollBit network failed verification (State of the Bots H1 2026). A user-agent string is a text field anyone can set, and Claude-User is a popular disguise precisely because many sites allow it.

Three checks catch most of it:

Origin. Anthropic's support page for its crawlers is the reference for current verification guidance. - Behavior. A "Claude-User" fetching /wp-admin, /.git/config or /.ssh/id_rsa is a vulnerability scanner wearing a badge. Most of the unreachable URLs in our logs are probes of this kind under AI-looking user agents, not real content gaps.
Timing of the check. Verify at the edge before serving, not in analytics afterward. Bots do not run JavaScript, so analytics never sees them anyway.

How to treat them

Claude-User is the agent that produces answers about you right now, so allow it, serve it something readable, and measure what it reads. Claude-SearchBot is how you appear in Claude's search results, costs almost nothing to serve, and should be allowed. ClaudeBot is a training crawl, and the decision there is a business one: allow, charge per fetch through a bot paywall, or disallow with the ClaudeBot token.

The same rule applies to all three: identify before serving. TollBit's Bot Index tracks the three agents across 10,000+ sites, and the bot paywall lets you set different terms for each, for example free for Claude-User and Claude-SearchBot, priced for ClaudeBot.

Frequently asked questions

Does blocking ClaudeBot remove my site from Claude's answers? No. ClaudeBot only feeds training. Answers to a user's question come from Claude-User fetching your page live, and search results come from Claude-SearchBot's index. Block all three and you disappear from Claude. Block only ClaudeBot and you keep both.

Why does Claude-User fetch robots.txt so often? Because it checks before it reads. Nearly 600 robots.txt fetches against 3.2K total reads on our site means it re-checks frequently rather than holding the file for long. The practical effect is that a change to your robots.txt reaches Claude-User fast.