Who’s visiting your site now: the types of AI visitors brands need to serve

Your customers are increasingly using AI tools to research and complete tasks on their behalf. This is causing bot traffic on the internet to explode, as these AI tools rely on bots and agents to fetch content from the web in real time to respond to user queries. The problem is that websites weren't built for these machine users. They can’t access your site and navigate it like humans. You need to serve them differently. If you don’t, they'll go to a third party or competitor to complete research or a task on behalf of their human user.

Like you serve your human audience in different ways with a desktop version of your site, a mobile version, or even a mobile app, this machine audience also needs a tailored experience. Different bots and agents operate differently and need separate experiences to achieve their goals on your site. Some are reading a page to answer a question. Some are working through several pages on your site to complete a task on behalf of a customer, like comparing products or making a purchase.

Now, AI is moving beyond simple search to interactivity and hyper-personalization as agents become the new visitor. That shift is the opportunity for brands. You don't serve every customer the same way, and your AI visitors are an audience that needs different experiences too.

There are three primary types of bots and agents visiting your site today, and each one has different needs.

1. Simple bots and crawlers

Simple bots and crawlers make up the majority of AI traffic to websites today. These are bots reading a page to answer a question. When someone asks an AI app how SUVs compare on range or whether a toy is safe for their child of a certain age, a bot fetches the relevant pages and pulls the facts it needs to generate an answer.

They need facts they can easily access, in text they can easily read. This audience needs:

Answers to the questions people are asking
Content that is loaded on the server, so it appears on the first fetch
All important information in a machine-readable text

Where most sites fall short on serving this audience

Most sites were built for human visitors, and a lot of details on your site may be inaccessible to a bot. Product specs, ingredients, directions, and reviews could all be details that load through JavaScript after the page renders.

Side-by-side comparison of a product page in a human browser versus what AI agents fetch

In this example, the bot can see that a section called “Directions” exists, but the content itself does not load. Content in PDFs and images is largely inaccessible. A bot may be able to see that a video exists on a page, but can’t tell you what the video is saying. Simple bots and crawlers need content in a format where they can actually comprehend the information. If your site is unable to provide the information to this audience, they will move on to another source to get the information they were looking for.

2. Computer-use agents

Computer-use agents are tools like Claude Code, Codex, OpenClaw, and Perplexity Computer that can work through several pages to complete a task on behalf of a human. They need to click around and interact with a site. They arrive at your site with a specific job to do on your customers’ behalf: research a product for a buyer, find a flight, identify the best deal, check for up-to-date information. We have seen a significant uptick in these agents since the beginning of the year across the 10,000 sites on TollBit.

An agent completing a task has to be able to navigate your website. These agents have different needs:

Interactions that behave predictably
The autonomy to navigate a site without getting stuck
High task completion rates on the workflows that your customers care about
Token efficiency, so each step costs less for your actual customer using the agent. Tokens are small chunks of text AI reads, and every token costs time and money for the AI end user. Non-optimized pages use more of them.

Where most sites fall short on serving this audience

A website that was designed perfectly for humans can be actively hostile to advanced agents. Banners, pop-ups, modals, and menus are built for the human browsing experience, and agents can’t easily navigate them. Flows that were built in JavaScript, such as booking flows, can cause an agent to fail at the task at hand. Even if the task doesn’t fail, the agent may run out of token budget or suspect it will and give up.

When an agent interacts with a site not designed for it they are slow, costly, or even fail to complete the task at all. Your customer asked their agent to get something done, but their agent was unable to complete that task on your site, meaning the customer's end goal was not reached, or the agent went elsewhere to complete it.

With computer use and advanced agents, there are a variety of different protocols that companies have to consider supporting for efficient site access. These include MCP, WebMCP, A2A, ACP, UCP, MPP, etc. New protocols are coming out regularly, and it can be difficult and expensive to invest engineering time for a growing but newer audience, especially when it’s still unclear which protocols will be popular.

3. Personal assistants

Personal assistants are the newest type of AI visitor that have been gaining traction lately. The assistants are advanced personalized agents that take action on behalf of a named customer. Meta’s Muse, Instinct, and Dots by OpenAI are some of the early examples of personal agents. These agents can have purchase intent. They know your customer’s preferences, and they can carry a wallet and login credentials that are provided by their end user. These agents are still very new, but it’s clear they will require hyper-personalization and interactivity.

Personal assistants need the services that any logged-in customer would need, but in a way that agents can use them, including:

Authentication: an agent can sign in securely on behalf of its user
Payments: an agent needs to be able to pay on behalf of its user
CLI and API access to your services
Hyper-personalization

Since this is such a new audience, many companies are still figuring out how to handle them. Some companies may want to make certain workflows off-limits to these agents for now. Websites may need to limit sensitive actions like applying for a loan or making a large purchase. Even simple actions behind a login for many sites today don’t often have rate limits, which can degrade site performance if left unchecked. There are a lot of considerations when serving this new audience.

Where most sites fall short on serving this audience

These types of visitors are still extremely new, but we are already seeing instances of sites struggling to serve these personal agents. A recent TechCrunch article highlighted that personal agents, including Meta’s Muse and Instinct, are getting turned away by websites. While some companies are intentionally blocking these agents, others that are trying to serve them have been unable to. The article highlights complaints on social media around Muse being unable to complete a purchase on Walmart's site. According to the article “a spokesperson for Walmart told us these weren’t intentional. In fact, Walmart is partnered with Muse and said it wants to be where its customers are, and that now includes being available through AI agent experiences.” Companies know they have to serve this audience, but they have no way for an agent to authenticate on a customer's behalf, and no route other than the pages built for people.

How to start serving these new visitors

Your website's layouts, design, and interactions were all built for people, and you shouldn’t have to rebuild it to also work for AI visitors. AI visitors need a site built for them, the same way you have a desktop site and a mobile site designed to meet your human audiences.

TollBit has been hosting agent-optimized sites for three years, enabling more than 10,000 of the world's largest websites to create a parallel site optimized for agents.

Brands can use TollBit Agent Sites to serve all of these different types of AI visitors, without tying up their entire engineering department. Agent Sites allow brands to serve these agents a machine-readable version of their website that allows agents to take actions and consume content, without affecting their traditional website for humans.

Here's how it works. Setup is seamless and typically can be done through a single engineering integration. AI traffic is routed to your agent site, and to the agents it still looks like your own domain. Your CDN and your human-facing tech stack stay the same. Your customers keep getting the same site, at the same URLs and the same speed.

For simple bots and crawlers, we convert all of your content into machine-readable formats. Content hidden by JavaScript can now be served on the first fetch, PDFs and presentations are converted to pages agents can parse, and videos can become transcripts. With our Agent Site CMS, marketing teams can easily create agent-only pages and can add FAQs and fact boxes to their existing agent pages. For computer-use agents, we create Interactive Agent Sites that remove the banners, pop-ups and modals that trip agents up, ensuring they are served an experience that allows them to efficiently complete their tasks. In our testing, agents using TollBit Agent Sites saw up to 35% faster task completion and a 100% task completion success rate. TollBit also supports emerging protocols as they arrive, from MCP and WebMCP to A2A and ACP, without additional engineering lift.

For personal assistants, we provide the same experience and extend the capabilities needed for them to complete their task on behalf of the human user. These include authentication, payments, CLI and API access, and we will continue to add hyper-personalized experiences as these agents mature.

Across all of these use cases of serving these different AI visitors, nothing changes for your human visitors. Your direct customers continue to receive the same experience on your main site.

The bottom line

You already customize your website for specific human visitors by creating desktop sites, mobile sites, and apps. Now is the time to do the same for your AI visitors. Each type of AI visitor has different needs; TollBit Agent Sites can help your company meet them.

As more of your customers send an agent instead of visiting themselves, brands that serve each type well will have an advantage over competitors that are slow to focus on this audience.

Reach out to us by emailing team@tollbit.com or visit https://tollbit.com/agent-site-for-brands/ to learn more!

You may also be interested in

February 12, 2024

The Sinkhole is Opening, Publishers Are the Start

The way we consume information on the internet is undergoing radical change. AI-powered bots and assistants, capable of scraping and synthesizing information from the open web, are shifting how we find answers and learn about the world. While convenient for users, this spells impending doom for businesses built on the old foundation of web traffic.

Read more

October 16, 2025

TollBit Enters Japan Through Strategic Partnership with Japan Business Press and BI.Garage

TollBit, the AI-era content monetization platform that enables publishers to monitor, manage, and monetize AI usage of their content, today announced its official entry into the Japanese market through a strategic partnership with Japan Business Press Co., Ltd. (JBP) and BI.Garage, Inc.

Read more

July 29, 2025

DataDome and TollBit Partner to Protect and Monetize AI traffic

The conversation around AI agents and website content has shifted dramatically. Across industries, businesses are grappling with a surge in AI-driven traffic, and determining what is malicious and what is valuable. Our customers have been asking: How do we control this new wave of traffic while unlocking its potential?

Read more