Build your own bot CDN or use TollBit?
You can build a bot CDN yourself: a Cloudflare Worker or Vercel middleware that matches AI user agents and serves them a lighter page. The first version takes a platform engineer a few days. The standing cost is the part to price: a user-agent list that drifts monthly, spoofing checks, per-platform rendering, a bot-only sitemap, a 404 fallback for cited pages, and a log pipeline so you can see any of it. TollBit is that work already running across 10,000+ sites. The honest comparison is about who carries the maintenance, not whether the first version is hard.
What the DIY version has to do
A bot CDN is the origin for AI agents on your own domain. Humans and Googlebot never touch it. To build one you need every item below, and the list is the specification whether you build or buy.
- Identify AI agents. Match user agents (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot, meta-webindexer, Amazonbot, Bytespider, and so on) and verify them against published IP ranges where the operator publishes them. In H1 2026, 12.7% of traffic claiming to be Claude-User was spoofed, so user agent alone is not identification.
- Keep the list current. Our own small site saw 28 distinct AI agents in 30 days, including single-digit visits from Amzn-User, Google-Agent and Manus-User, none of which a rule set written a year ago would name. New ones arrive most months.
- Render for each class of visitor. Retrieval bots want slim HTML. Fetch APIs want Markdown. Computer-use agents need forms and buttons preserved. One template does not fit three visitors.
- Serve a bot-only sitemap and robots.txt. So agent-only pages are discoverable, and so expired URLs leave the sitemap agents read without touching the human one.
- Handle the citation click. When an assistant cites a page only agents can see and a human clicks it, the click must land on a real page, never a 404.
- Log and classify. Bots do not run JavaScript, so Google Analytics never sees them. You need server-side logging, user-agent classification, success versus blocked, reads by page, and AI referrer segmentation.
- Keep the facts identical. Same prices, same claims, same stock as the human page, on every deploy. Same facts, cleaner format, bots only.
What you save by building
Vendor cost, and a vendor in the request path. You keep complete control of rendering logic and of where logs live. If you run one platform, have a platform team with capacity, and your agent surface is a few dozen pages, this is a reasonable choice.
What you own by building
The maintenance. Items 1, 2 and 6 never finish. A rule set written in spring misses the agents that launch in autumn, and misses spoofed traffic from the start. And your logs only show your own domain, so you can see what agents read from you but not what they read about you elsewhere, which is where most of the reading happens.
What TollBit adds beyond the proxy
TollBit has operated agent sites for about three years and sees 200M+ AI bot fetches a day. The items above are the product, plus three things a single-site build cannot have:
- Network identification. Agent classification and spoof detection informed by traffic across 10,000+ sites, updated as new agents appear.
- Network demand. Which questions agents carry into your category, from reads on the sites that answer them, not only your own.
- A CMS for the bot version. Marketing edits what agents see, publishes agent-only pages and reviews per-platform versions without an engineering ticket.
Setup is two IT changes: forward server logs and point a tollbit.yourbrand.com subdomain, with a routing rule on whatever CDN or WAF you already run (Akamai, Cloudflare, Fastly, CloudFront, DataDome, Imperva, HUMAN).
Side by side
| Build it yourself | TollBit | |
|---|---|---|
| Time to first version | Days | Two IT changes |
| Agent list upkeep | Yours, monthly | Ours, network-informed |
| Spoof detection | Per published IP range, if you add it | Included |
| Per-platform rendering | Build per class | Included, editable in a CMS |
| Sitemap, robots.txt, 404 fallback | Build each | Included |
| Analytics | Your domain only | Your domain plus network reads |
Frequently asked questions
Can I start DIY and move later? Yes. The routing rule is the same in both cases: match identified AI agents, send them to an origin. Moving means changing where the rule points and forwarding logs.
Does a vendor in the request path slow my site for humans? No. Humans are never routed to the bot CDN. Only identified AI agents take that path, and the human origin sees less load as a result.
Which is right for a publisher versus a brand? Publishers with engineering teams often build the first proxy and outgrow it when they want pricing, licensing and analytics. Brands rarely want to own the agent list at all. Both arrive at the same requirements.