What is a bot CDN?
A bot CDN is the origin server for AI agent traffic on your domain. A conventional CDN sits in front of your site and caches copies of your pages so humans around the world load them faster. A bot CDN sits beside it and answers the requests from AI agents, serving them a rendering built for machines instead of the page built for browsers.
The two do not compete. Your existing CDN keeps doing its job for humans. It gains one rule: when the visitor is an identified AI agent, hand the request to the bot CDN.
CDN and bot CDN, side by side
| Conventional CDN | Bot CDN | |
|---|---|---|
| Who it serves | Humans in browsers, Googlebot, Bingbot | AI agents: crawlers, retrieval bots, computer-use agents |
| What it does | Caches and accelerates your pages | Is the origin for the agent-facing version of your pages |
| What it returns | The human HTML, as built | Server-rendered text or slim HTML, with the same facts |
| What it knows | Edge hits, bandwidth, cache ratio | Which agent read which page, success or failure, referrals |
| Who decides the content | Your CMS | Your team, through the Agent Site CMS, page by page |
A CDN optimises delivery of one rendering. A bot CDN produces a second rendering for a second audience.
Why a cache is not enough for AI agents
Caching assumes the visitor wants the same bytes a browser wants. AI agents do not. A retrieval bot answering a user's question loads your page fully into a context window, and every script tag, menu item and cookie banner in that HTML costs it tokens before it reaches the first useful sentence. Across ten publisher articles on our network, the machine-readable rendering was 56% to 97.5% smaller than the human HTML.
Caching also assumes the visitor will come back for the cached copy. AI applications cannot cache the internet. Grounding and retrieval re-fetch pages at answer time, and agents do not get tired, so the same page is pulled again and again. Our network of 10,000+ sites sees 200M+ AI bot fetches a day. The question is not how to serve those fetches faster. It is what to serve them.
How bots get to it
Two changes, both inside IT's existing tools:
- A subdomain. You create
tollbit.yourbrand.comand point it at the bot CDN with a CNAME record. - A routing rule. In your CDN or WAF, a rule matches the user agents of known AI agents (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Amazonbot, meta-webindexer and the rest) and forwards their requests to the subdomain.
The bot asks for yourbrand.com/page. It receives the agent-facing version of that page. It never sees the subdomain, and the URL it cites is your real URL. This works on top of Akamai, Cloudflare, Fastly, CloudFront, DataDome, Imperva, HUMAN and WordPress VIP, because every one of them already has a way to route by user agent.
The bot CDN also serves the robots.txt and sitemap that bots read, which is how pages that exist only for agents become discoverable.
Why Googlebot and Bingbot are never routed
Search engines index the human page and rank it, and your SEO depends on them seeing exactly what a human sees. So the routing rule excludes them, by design. A bot CDN serves the agents that search engines were never built to handle: retrieval bots that fetch at answer time, AI search indexers, training crawlers and computer-use agents completing tasks. Google and Bing continue to reach your origin, or your human CDN, unchanged.
What a bot CDN sees that nothing else does
Because every AI fetch passes through it, a bot CDN holds the only complete record of AI reading on your site. Google Analytics runs in JavaScript, and bots do not execute JavaScript, so it records none of this. On tollbit.com between September 15 and October 14, 2026, our own logs show about 44K AI crawl attempts from 28 distinct agents, with meta-webindexer, ChatGPT-User, ShapBot, Claude-User and Amazonbot the five largest readers. That picture exists in server logs and nowhere else.
Where TollBit fits
TollBit is a bot CDN. It began as the bot CDN and paywall for 10,000+ publisher sites, where the job was to see which AI bots read your pages, set terms and get paid. The same rail now runs the agent-facing version of brand and commerce sites. The Agent Site CMS sits on top of it, and the analytics come out of the same logs.
Frequently asked questions
Do I have to replace my CDN? No. The bot CDN sits beside it. Your CDN adds one routing rule for AI user agents and continues to serve humans and search engines as before.
What happens if a new AI agent appears that my rule does not match? It reaches your human site as it does today. The user-agent list is maintained on the bot CDN side and updated as new agents declare themselves.
Does the bot CDN slow down AI agents? It speeds them up. The agent-facing rendering is a fraction of the size of the human page, and the bot CDN serves it without the scripts and assets a browser would also load.