How to route AI bot traffic to a subdomain
Routing AI bot traffic to a subdomain takes two changes, both inside tools your IT or security team already operates. First, create a subdomain such as tollbit.yourbrand.com and point it, with a CNAME record, at the bot CDN that will serve agents. Second, add a rule in your CDN or WAF that matches the user agents of known AI agents and forwards their requests to that subdomain. A third step, forwarding server logs, gives you the measurement.
No code changes on your site. No CMS plugin. Humans, Googlebot and Bingbot are never matched by the rule and never see the subdomain. Neither, in practice, do the bots.
Change 1: the CNAME
Create a DNS record for the subdomain and point it at the hostname the bot CDN gives you. Any name works. tollbit.yourbrand.com is the convention on our network, agents.yourbrand.com is as good. The subdomain is on your domain, so the agent-facing pages are your pages, and the URLs agents cite remain yourbrand.com/....
A TLS certificate for the subdomain is issued on the bot CDN side. This change usually takes a few minutes of a DNS administrator's time plus propagation.
Change 2: the routing rule
Every modern CDN and WAF can route by request header. The rule says: if the User-Agent header matches one of these AI agents, send the request to the subdomain's origin instead of the normal one. Everything else is untouched.
| Platform | Where the rule goes |
|---|---|
| Akamai | A Property Manager rule with a user-agent match and an origin override |
| Cloudflare | A rule or Worker that rewrites the origin for matched user agents |
| Fastly | VCL or Compute logic that selects a second backend on user-agent match |
| Amazon CloudFront | A Lambda@Edge or CloudFront Function that changes the origin |
| Imperva, DataDome, HUMAN | A bot-management policy that classifies AI agents and forwards rather than blocks |
| WordPress VIP | Edge configuration through the platform |
The match list is the hard part to maintain alone. GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Amazonbot, meta-webindexer, meta-externalagent, Bytespider, CCBot, Google-Extended and more today, with new agents declaring themselves regularly. On tollbit.com alone we saw 28 distinct AI agents in the past month. A bot CDN maintains this list as part of the service, so the rule does not depend on someone editing a regex when a new agent appears.
The rule excludes Googlebot and Bingbot explicitly and does not match ordinary browsers. Search indexing and human traffic are unchanged by design.
Change 3: forward the logs
The bot CDN sees every request it serves. Your origin and your CDN see the requests that did not match the rule, including blocked and failed AI fetches, and the humans who arrive from AI answers with a referrer such as chatgpt.com. Forwarding your CDN or server logs to the same analytics gives one view: which agents fetched which pages, which succeeded, which were blocked, which URLs they asked for that do not exist, and which humans came back. Google Analytics shows none of the agent side, because agents do not run its JavaScript.
Log forwarding is a standard feature on every platform above (log streaming, log delivery, Logpush and their equivalents).
What the bot experiences
An agent requests https://yourbrand.com/products/widget. Your CDN matches its user agent and fetches the response from the subdomain's origin instead of yours. The bot CDN fetches the human page from your origin, transforms it into the agent-facing rendering, and returns it. The agent receives a 200 from yourbrand.com/products/widget with the agent-facing content. It never sees tollbit.yourbrand.com, never follows a redirect, and cites your real URL.
Because the bot CDN fetches from your origin at request time, the agent version carries your current prices, stock and copy. There is no sync and no second copy of your site to maintain.
Three questions security teams ask
- Does this bypass bot management? No. The bot manager still classifies every request. The change is the action for AI agents: forward instead of block or serve the human page. Unknown and malicious automation is handled as before.
- Can a bot spoof its way in? Any request can claim any user agent, and in H1 2026 12.7% of Claude-User traffic on our network was spoofed. Routing a spoofed request to the agent site gives it a text rendering of a public page, nothing more. Where verification matters, published IP ranges and request signatures are checked on the bot CDN side.
- What if the subdomain is down? Set the rule to fail open to your normal origin. The agent gets the human page, as it does today.
Where TollBit fits
TollBit is the bot CDN at the other end of the CNAME. It maintains the AI user-agent list, is the origin for identified agents on your domain, serves the robots.txt and sitemap they read, and turns the forwarded logs into per-agent, per-page analytics. The integration is the two changes above and works on Akamai, Cloudflare, Fastly, CloudFront, DataDome, Imperva, HUMAN and WordPress VIP.
Frequently asked questions
How long does setup take? The DNS record and the routing rule are usually an afternoon for a team that already administers the CDN. Log forwarding is a configuration change on the same platform.
Do I need to change anything in my CMS or codebase? No. The bot CDN reads your pages over HTTP like any visitor. Your CMS, front end and deployment pipeline are not involved.
Will Google see a different page and penalise me? Googlebot and Bingbot are excluded from the rule and continue to reach your normal origin. They never see the agent-facing rendering.