ShapBot, Bytespider, CCBot and the AI crawlers you have not heard of

In total, 28 distinct AI agents read tollbit.com between Sep 15 and Oct 14 2026, and the ten below account for close to 9K reads between them. Each is listed with what it did on our site and what is publicly documented about its robots.txt behavior. Where the operator has not published something, we say so.

Crawlers for newer search and answer engines

ShapBot (3.7K reads here, third-largest of all AI agents on our site). Its user agent gives no operator name and links to no documentation page, so who runs it is not declared. Its pattern on tollbit.com is a broad, repeated crawl, consistent with building an index that other people's agents query. Its robots.txt handling and IP verification are not publicly documented in a form we can point to.

Shap-User (0.9K). The user-triggered counterpart, following the "-User" convention. It fetches specific pages rather than sweeping, which fits retrieval on behalf of a developer's application. Treat it as RAG traffic. Documentation is as thin as ShapBot's.

YouBot (0.4K). You.com's crawler, feeding its search and assistant products. You.com publishes a page describing YouBot and the token to use in robots.txt. Behavior here was light and indexer-shaped.

DuckAssistBot (0.5K). DuckDuckGo's fetcher for DuckAssist, the AI answers that appear in DuckDuckGo search results. DuckDuckGo documents it, states that it honors robots.txt, and publishes the IP addresses it uses. DuckDuckGo as a whole sent 126 referrals to tollbit.com in the period, search and assistant mixed.

cohere-ai (0.4K). Cohere's crawler. Cohere builds enterprise models and a RAG platform, and the token appears in robots.txt files across the network. Cohere has published little about the agent's purpose, IP ranges or robots.txt handling, so treat the token as unverified identity.

MistralAI-User (0.34K). Mistral's user-triggered fetcher, behind web access in Le Chat and Mistral's agent products. Mistral documents the token. Its pattern here was retrieval-shaped: specific pages, no sweep.

Training and archive crawlers

Bytespider (1.0K). ByteDance's crawler. ByteDance has published very little about it, and in 2024 several publishers and bot-management vendors reported it fetching paths disallowed in robots.txt. No published IP list that we are aware of. Which ByteDance products the crawl feeds is not publicly documented. It behaves like a training crawl here: broad, indifferent to what people are asking about.

CCBot (1.2K). Common Crawl's crawler. Common Crawl is a nonprofit that publishes a free monthly archive of the web, and that archive is one of the most widely used training sources for language models. Its FAQ states that it honors robots.txt and nofollow, and the token is CCBot. Blocking it removes you from the archive and from every model trained on it, which is the one robots.txt decision with the longest reach.

Google tokens that are not what they look like

Google-Extended (0.35K). This is a robots.txt token, not a crawler. Google's documentation says Google-Extended controls whether pages Googlebot already crawls may be used for Gemini training and grounding, and that it has no user-agent string of its own. The 350 fetches in our logs carrying it as a user agent are therefore worth checking against Google's published IP ranges before being counted as Google.

Google-NotebookLM (50 reads). Google's user-triggered fetcher for NotebookLM, which fetches a page when a user adds it as a source. Google lists it among its user-triggered fetchers, which Google says generally ignore robots.txt because a person requested the page. 50 fetches means 50 people put a tollbit.com page into a notebook.

How to treat the long tail

The pattern is the same as for the large operators. Allow indexers and user-triggered fetchers, decide on training crawls, and verify every token against a published IP range where one exists. Where none exists, the token is a claim, not an identity. TollBit's Scraper Index tracks the undocumented end of this list across 10,000+ sites, and the bot paywall applies per-agent terms to any of them.

Frequently asked questions

Should I block every crawler I do not recognize? No. Block what you can identify as training you do not want to support, allow what feeds search and answers, and verify everything else before deciding.

Why does a token I disallowed still show up in my logs? Three reasons: the operator does not honor robots.txt, the request is a user-triggered fetch the operator exempts, or the user agent is a scanner copying a token. Most unreachable URLs in our logs are vulnerability probes under AI-looking user agents, which fits the third.