Why is a coding agent reading my website?

Because a developer told it to. Coding agents such as Claude Code fetch web pages when a developer asks them to read documentation, pull a dataset or check an API, and on tollbit.com the user agent claude-code fetched our State of the Bots report about 1,350 times between Sep 15 and Oct 14 2026. Nobody planned for that visitor. It is the leading edge of the computer-use class, and it is already here.

What a coding agent is, as a visitor

Claude Code is Anthropic's command-line coding agent. A developer types a goal, the agent reads files, runs commands and, when it needs something from the web, fetches a page and converts it to text. In our logs it declares itself as claude-code. Other coding agents (OpenAI's Codex CLI, Google's Gemini CLI, editor-based agents) do the same job, and their identification varies: some carry a product token, some use a generic HTTP library string. Not every coding agent announces itself.

Three things distinguish it from a crawler or a chat assistant's fetcher:

A human is in the loop, with a goal. The fetch is one step in a task, not a sweep.
It has a token budget. Every page it reads is paid for in context window, so it wants the fact, not the page furniture.
It does not render. It reads server-rendered HTML and converts it. A JavaScript app shell gives it a few dozen words.

What it read on our site

About 1,350 fetches of one URL, the State of the Bots report. The report is a public dataset: 987B+ website visits analyzed, 22B+ AI bot scrapes, per-agent and per-region figures. Developers were piping it into coding sessions, to parse the numbers, build a chart, compare the figures against their own server logs or write a classifier for AI user agents.

Reading documentation is the normal coding-agent pattern: an engineer integrating something asks the agent to read the docs rather than reading them personally. Compared with Claude-User, which read 3.2K pages spread across the site with robots.txt first, claude-code read one page over and over. Same operator, different visitor.

Why this is the leading edge of computer-use agents

We describe three classes of AI visitor: simple bots and crawlers, computer-use and research-and-task agents, and personal assistants. A coding agent is the second class before it has a browser. It navigates toward a goal, follows links when a page points elsewhere, retries when a fetch fails, and stops when the task is done. The questions it asks of your site are the usability questions: can I get the fact in one fetch, is it in a form I can parse, does the URL I was given exist.

The volume is small today. It will not stay small. Usage-based billing for agents means every wasted fetch lands on the developer's bill, and developers route their agents toward sites where tasks complete cheaply.

What to serve them

Text first. The number, the date, the unit, in the HTML. A chart image with no table behind it is a dead end.
Tables as tables. HTML tables or Markdown tables, not screenshots of spreadsheets.
Data next to the story. A CSV or JSON link beside the headline figure saves the agent a scrape and saves you the load.
Stable URLs. A coding agent given a URL by a developer fetches exactly that URL. A redirect maze or an expired path is a failed task.
Docs with complete examples. Full request and response, not a fragment with "..." in it.
No PDF-only publishing. A PDF is readable but costly to extract. Serve the HTML version too.

An agent site does this without touching the human page: the same report, served to bots as slim semantic HTML or Markdown, with the data tables intact, and the agent-only pages discoverable through the sitemap bots read.

Frequently asked questions

Should I block coding agents? Only if you block developers. A coding agent reading your docs is an engineer evaluating or integrating your product, which is the highest-intent visit a developer-facing site gets. Measure it, serve it well, and treat it as retrieval traffic.

Is claude-code the same as Claude-User? No. Claude-User is the fetcher behind Claude's chat and search products, and it fetches robots.txt before content. claude-code is a developer tool fetching whatever a developer asked for. Both are Anthropic agents, but they are different visitors with different patterns.