How do I measure AI bot traffic on my website?
In your server logs, not in Google Analytics. AI bots do not execute JavaScript, so tag-based analytics never records them, and the only complete record of an AI visit is the request line your server or CDN wrote when it fetched the page.
Measurement has four parts: classify the user agents, count successful reads by page, separate reads from blocks, and match reads against the human referrals that AI answers send back.
Step 1: get the logs out
Forward origin or CDN logs (Akamai, Cloudflare, Fastly and CloudFront all export them) to wherever you will analyze them. You need the user agent, IP, path, status code and timestamp. That is enough to reconstruct every AI read of your site.
The median North American site in our H1 2026 State of the Bots report saw 1 AI bot visit for every 93 human visits, and the European median was 1 in 33. On tollbit.com, from Sep 15 to Oct 14 2026, we logged about 44K AI crawl attempts from 28 distinct AI agents.
Step 2: classify the agents
Group user agents by operator and by purpose, because the purpose decides what a read means.
- Training bots (GPTBot, ClaudeBot, CCBot, Bytespider) ingest archives. A read is a copy taken.
- AI search indexers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer) build the index an assistant searches. A read is a chance to be found.
- Retrieval bots (ChatGPT-User, Claude-User, Perplexity-User) fetch a page because a person asked a question right now. A read is your page inside an answer.
Verify identity rather than trusting the string. Most operators publish IP ranges, and a fetch claiming an agent from an unlisted IP goes in a separate bucket.
Step 3: count reads and blocks by page
A read is a fetch that returned 200. A block is a 403 or a 402. A gap is a 404 or a 5xx. Keep all three.
The top of our own list for the period:
| Agent | Successful reads |
|---|---|
| meta-webindexer | 16.3K |
| ChatGPT-User | 10.5K |
| ShapBot | 3.7K |
| Claude-User | 3.2K |
| Amazonbot | 2.9K |
| meta-externalagent | 2.3K |
| PerplexityBot | 1.9K |
| ClaudeBot | 1.8K |
Then sort by page. Our most-read page by AI is the homepage, then the State of the Bots report, then robots.txt, then the bot paywall page. The coding agent user-agent claude-code fetched the State of the Bots report about 1,350 times, more than any other page: developers are piping the report into their coding agents. None of that appears in a pageview report.
Step 4: match reads to referrals
Human clicks from AI answers arrive as ordinary traffic with a referrer: chatgpt.com, claude.ai, perplexity.ai, gemini.google.com, copilot.microsoft.com. Segment them. Then divide reads by referrals.
For tollbit.com in the period it was 47 AI referrals against roughly 44K reads, about 1,000 to 1 (ChatGPT 37, Claude 7, Perplexity 3, plus 126 from DuckDuckGo with search and assistant mixed). Across European sites in our H1 2026 report, the scrape-to-referral ratio ran from 150:1 in Q1 to 227:1 in Q2. AI reads you far more than it sends anyone to you.
What to look at first
- Unreachable pages. Which URLs did agents request and fail to get? Look for JavaScript-only pages, PDFs and expired URLs still in the sitemap. Filter the noise first: most "unreachable" URLs in our logs are vulnerability-scanner probes (/.git/config, /wp-admin) under AI-looking user agents, not content gaps.
- Read a lot, cited little. Pages with many retrieval reads and no referrals shape answers without credit. Citations run on the order of 5–15% of what an assistant reads. Tighten those pages first.
- robots.txt as the top fetch. OAI-SearchBot's most-fetched URL on our site is robots.txt (about 1,000 fetches), and so is Claude-User's (nearly 600). Agents check permissions constantly, so what that file says matters.
TollBit Analytics does these four steps from forwarded logs for the 10,000+ sites on our network: agent, purpose, success rate, reads by page, referrals.
Frequently asked questions
Can I do this in Google Analytics or a tag manager? No. Tags run in the browser, and AI bots do not run a browser. Server logs, CDN logs or a WAF are the only places an AI read exists.
How do I tell a real AI agent from a spoofed one? Check the source IP against the operator's published ranges and, where offered, reverse DNS. The user-agent string alone is a claim, not an identity. In H1 2026, 12.7% of traffic claiming to be Claude-User was not.
How often should I look? Weekly is enough for trends, since recrawl cadence varies by platform from hours to weeks. Look daily when you change something, such as publishing an agent-only page or editing robots.txt.