Glossary of the agent web

These are the 22 terms we use for the web as AI agents use it. The thesis under all of them: AI is its own visitor, with its own needs, and deserves its own experience the way mobile did.

Visitors

AI visitor. Any AI system that fetches a page on your site: a crawler, a retrieval bot, a search indexer, a task agent or a personal assistant. Distinct from a human arriving via an AI answer.

Simple bots and crawlers. The largest class of AI visitor today. They read server-rendered text, do not run JavaScript, and want availability, pre-computed answers and a sitemap. Their question of a site: are you readable?

Computer-use agent (advanced agent). An agent that navigates toward a goal: compares, fills forms, adds to cart, completes a task. It runs on a token budget, so it wants slim formats and stable paths. Its question: are you usable?

Personal assistant. An agent acting for one identified person, with their preferences, logins and payment methods. Meta launched Muse in September 2026, the same month Resy began deactivating accounts whose assistant hit its site about 200 times an hour. The groundwork (authentication, payments, APIs) is still being laid.

Training bot. A crawler that ingests pages to train a model, such as GPTBot, ClaudeBot or CCBot. One-off value: a page is worth something once.

Retrieval (RAG) bot. A fetcher that reads a page live to answer a user's question, such as ChatGPT-User, Claude-User or Perplexity-User. Recurring value, because the page is loaded into a context window every time the question is asked. RAG cannot cache the internet.

AI search indexer. A crawler that builds the index an assistant searches before it fetches, such as OAI-SearchBot, Claude-SearchBot, PerplexityBot or meta-webindexer. Sitemap-driven, periodic, broad.

Infrastructure

Agent site. The agent-facing version of your website: same products, prices and logic, different rendered surface, served on your own domain to identified AI agents. The mobile site of the agent era.

Bot CDN / agent CDN. The reverse-proxy origin for AI bot traffic on your domain. A CDN caches pages for humans. A bot CDN is where identified agents are routed by a subdomain and a CDN or WAF rule, while humans, Googlebot and Bingbot are never routed.

Agent web stack. The agent-era counterpart of each layer of the human web: server-side AI analytics for Google Analytics, an agent site for the CDN and CMS surface, agent advertising for the ad server, machine payments for checkout, network-wide intelligence for keyword tools.

Agent-only page. A page that exists only in the sitemap bots read. Used for comparisons, FAQs, spec sheets and "is X safe" questions. Same facts as the human site, cleaner format, bots only.

Bot-only sitemap. The sitemap served to AI agents, which differs from the one served to Google: it lists agent-only pages, drops expired URLs and can vary per platform.

Citation fallback. When an AI answer cites an agent-only page and a human clicks, the click lands on the closest real page on the human site, never a 404.

Crawl success. The share of AI fetch attempts that return the content requested. Blocked, challenged, timed-out and JavaScript-only responses count as failures. The first number to look at in server logs.

Measurement

AI read (AI impression). One AI agent fetched one page, successfully. Observed in server logs, never in page analytics, because bots do not execute JavaScript. Our network observes 200M+ a day across 10,000+ sites.

AI referral. A human arrived on your site from an AI answer, identified by referrer (chatgpt.com, claude.ai, perplexity.ai and others). On tollbit.com, reads outnumber referrals about 1,000 to 1.

Share of reads. Your pages' share of AI reads on a topic, measured across the network's logs. The leading indicator. Share of voice, from sampled prompts, trails it, because an assistant cites 5 to 15% of what it reads and advanced agents often cite nothing.

Payments and protocols

Machine payments. An agent paying for access or goods in the course of a task, with no human at a checkout page. Per-fetch payment for content is live today. Agent checkout for goods is the next layer.

HTTP 402. The Payment Required status code, reserved in the HTTP specification since the 1990s and now used to quote a per-fetch price to a bot before content is served.

Bot paywall. Identify the agent, apply your terms, charge or refuse, serve. 2.6B+ AI bots were directed to the TollBit Bot Paywall in H1 2026. It sits behind existing bot management.

Licensed RAG. Retrieval at answer time that is paid for and sanctioned: a developer searches, sees a price, fetches licensed content in a bot-ready format. Recurring revenue, unlike a one-off training license.

MCP. Model Context Protocol, an open standard that lets an AI application connect to tools and data through servers that describe what they offer. Started by Anthropic in late 2024, now broadly adopted and under open governance.

WebMCP. An early browser proposal from Google and Microsoft engineers that lets a web page declare tools an in-browser agent can call directly, instead of clicking through the human interface. Unsettled, and worth watching.

NLWeb. Microsoft's open project for giving a website a natural-language interface over its own content, built on Schema.org data, with each NLWeb endpoint also acting as an MCP server.

Frequently asked questions

What is the difference between an AI read and an AI referral? A read is the agent fetching your page. A referral is a human arriving after an answer cited you. Reads come first and far more often, so they are the leading measure.

Is an agent site the same as GEO? No. GEO optimizes presence in generated answers. An agent site sits underneath: it makes your site readable and usable to agents in the first place, then measures what they read in the same logs.