Does robots.txt actually stop AI crawlers?
Partially. robots.txt is a request, not a lock. It stops the crawlers that choose to honour it, and our logs, 200M+ AI bot fetches a day across 10,000+ sites, show that honour is uneven. In H1 2026 we detected 22B+ AI bot scrapes, and 1.9B+ of them ignored robots.txt (State of the Bots H1 2026). Declared training crawlers mostly comply. User-triggered fetchers operate under different rules. And a share of AI traffic arrives with a spoofed or absent identity, which robots.txt cannot touch by definition.
If the goal is control, robots.txt is the polite first layer. Enforcement happens at the server.
The three kinds of AI bot, and how each treats robots.txt
| Kind | Declared agents | What it does | robots.txt behaviour |
|---|---|---|---|
| Training crawlers | GPTBot, ClaudeBot, CCBot, Bytespider, meta-externalagent | Bulk-ingest archives for model training | Major Western crawlers generally honour a disallow. Compliance varies elsewhere |
| AI search indexers | OAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer | Build the index an assistant searches | Generally honour robots.txt. On tollbit.com, robots.txt is OAI-SearchBot's most-fetched URL, about 1,000 fetches in 30 days |
| User-triggered fetchers | ChatGPT-User, Claude-User, Perplexity-User, Google-Agent | Fetch a page live because a person asked | Treated by operators as user requests, not crawling. robots.txt is often not the governing rule |
The practical consequence: a disallow reduces training-crawler traffic, but the fetches that power live answers about you can continue, and undeclared scrapers ignore the file entirely.
Google has said so out loud
In April 2026 Google documented Google-Agent, the fetcher behind Project Mariner and its agent products. The documentation states that, like all user-triggered fetchers, it generally ignores robots.txt, because requests originate from a person's instruction rather than autonomous crawling. Google publishes its IP ranges and a verifiable reverse DNS pattern, so the agent is identifiable. It does not violate robots.txt. It opts out of it.
What the H1 2026 numbers show
- 1.9B+ AI bot scrapes ignored robots.txt, out of 22B+ detected.
- robots.txt was ignored 2.8 times more often in the EU than in North America.
- 12.7% of traffic claiming to be Claude-User was spoofed: a user agent string anyone can type, with none of Anthropic's published IP ranges behind it. A robots.txt rule for Claude-User does nothing to the spoofed share, and a block on it punishes the real one.
None of this is visible in Google Analytics, because bots do not run JavaScript. It is visible in server logs.
The special case of Google search
Google's AI features draw on infrastructure shared with search crawling. Blocking Googlebot removes you from search itself. The separate Google-Extended token governs some AI training use but not every AI feature. There is no robots.txt configuration that removes a site from AI answers while keeping its search presence fully intact.
What a robots.txt rule can still do
It declares intent to the operators who listen, and it is cheap. A minimal file that allows search indexers and disallows the training crawlers looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Googlebot
Allow: /
Treat it as the written policy. The enforced policy lives elsewhere.
What actually enforces a policy
- Server-side identification. User agent plus published IP ranges and request verification, checked before content is served. This is where spoofed Claude-User traffic is separated from the real thing.
- Edge responses per bot and per path. Serve, refuse (403), or price (HTTP 402 Payment Required).
- Charging as the middle path. A 402 with a per-fetch rate turns bot traffic from a cost into a market. Bots that value the content pay and read. Bots that do not, stop. In H1 2026, 2.6B+ AI bots were directed to the TollBit Bot Paywall.
- Monitoring undeclared traffic. Evasion exists. A control stack that only trusts self-identification undercounts it.
Akamai, DataDome, HUMAN, Imperva and Cloudflare do the identification. TollBit is where identified AI bots are routed and where serve, refuse or price is applied and logged.
Frequently asked questions
Should I block AI bots or charge them? Blocking is right for content you never want in AI systems, and it works on declared traffic: after a block, declared AI fetches fall quickly, and a few weeks later so does presence in AI answers. Assistants stop reading you, then stop mentioning you. Charging preserves presence while ending free extraction.
Does adding GPTBot to robots.txt remove my content from ChatGPT? It stops the training crawler, and OpenAI honours it. Live retrieval by ChatGPT-User on a user's request is governed separately, so answers about you may still fetch and read your pages.
Do llms.txt or Markdown versions help control bots? No. In observed network logs, retrieval bots overwhelmingly fetch ordinary HTML pages, and alternative bot-facing files see marginal fetch volume. They are not a control surface. Control is identification plus a server response.