What is a bot paywall?
A bot paywall is a layer at the edge of a website that identifies AI bots, applies the owner's terms to each one, charges the ones with a price, and serves the content in a format bots can use. Humans never see it, because humans are never routed to it. In H1 2026, 2.6B+ AI bots were directed to the TollBit Bot Paywall across 10,000+ sites.
It does for AI visitors what a subscription paywall does for readers, except the terms can be checked and settled in the same request.
The four steps
1. Identify. Every request carries a user agent and comes from an IP address. The paywall matches the user agent against known AI agents (ChatGPT-User, ClaudeBot, PerplexityBot, Amazonbot, meta-webindexer, GPTBot and the rest), checks the IP against the operator's published ranges, and separates spoofed traffic from the real thing. In H1 2026, 12.7% of traffic claiming to be Claude-User was spoofed.
2. Apply terms. The owner's rate card says what each class of bot gets on each part of the site: serve free, serve for a per-fetch price, or refuse. Training crawlers, retrieval bots and AI search indexers can each get different terms. So can named agents and paths.
3. Charge. For bots with a price, the paywall returns HTTP 402 Payment Required with the rate. The bot's operator pays through the rail and re-requests. Bots whose terms say refuse get a 403. Bots whose terms say free pass through.
4. Serve. Paid and allowed bots receive the content in a bot-ready format, Markdown or slim HTML, with licensing metadata attached. Every fetch is logged with agent, page, status and price, into the same analytics that show all AI reads on the site.
Human paywall versus bot paywall
| Subscription paywall | Bot paywall | |
|---|---|---|
| Visitor | A person in a browser | An AI agent making an HTTP request |
| Identification | Login, cookie | User agent, IP range, request signature |
| Terms | Per subscriber plan | Per bot class, named agent, path, freshness |
| Price event | Monthly | Each fetch |
| Format served | The web page | Markdown or slim HTML |
| Who sees it | Everyone who hits the meter | Only identified AI bots |
A site can run both. They do not interact. Googlebot and Bingbot are never routed to the bot paywall, by design, so search indexing is unaffected.
What the bot gets, and why that matters
A bot that pays gets a clean copy of the page: the content, without scripts, trackers, ads and layout markup. In our public comparison a Forbes article dropped from 1,201 KB of HTML to 41 KB of Markdown, and across ten articles the reduction ran from 56% to 97.5%. For the AI operator that is a cheaper, faster, licensed input than a scraped page. For the site owner it is less origin load, since the bot traffic is served from the agent origin rather than the human one.
It also preserves presence: a blocked site disappears from answers, a paying bot reads the current page.
Works with existing bot management
A bot paywall does not replace Akamai, Cloudflare, Fastly, DataDome, Imperva or HUMAN. Those products detect and classify bots at the edge. The bot paywall is where identified AI bots are routed, and where serve, charge or refuse is applied. In practice it is a routing rule in the existing CDN or WAF that sends AI user agents to a TollBit subdomain on the owner's domain, plus log forwarding. Two IT changes.
Where it sits in the stack
The bot paywall is the oldest piece of what TollBit now calls the agent web stack. The same rail that returns 402 to a training crawler is the rail that serves an agent-facing version of a product page, lists agent-only pages in a bot-only sitemap, and measures pickup in server logs. Publishers started with terms, brands start with the agent site, and the identification step is shared.
Frequently asked questions
Does a bot paywall block Google? No. Googlebot and Bingbot are excluded from routing, so search indexing continues exactly as before. The paywall applies only to identified AI agents, and Google-Extended, Google's AI training token, can be given its own terms.
What happens to a bot that will not pay? Whatever the owner's terms say. It can be refused with a 403, or served a limited free version, or allowed through if the owner values the presence more than the fee. Most owners set different answers for different bot classes.
Can I use the paywall only to see traffic, without charging? Yes. The rate card can be changed at any time without touching the site. What you learn first is who reads you, what they read, and how much of it Google Analytics never saw.