What are meta-webindexer and meta-externalagent?
The largest AI reader of tollbit.com between Sep 15 and Oct 14 2026 was not OpenAI or Anthropic. It was meta-webindexer, Meta's web indexer, with 16.3K successful reads, more than ChatGPT-User (10.5K) and more than Anthropic's three agents combined. Meta's training crawler, meta-externalagent, added 2.3K. Meta is the biggest AI visitor most site owners have never looked at.
Meta's three AI agents
| Agent | Job | robots.txt, per Meta's documentation |
|---|---|---|
| meta-webindexer | Crawls to build the web index behind search in Meta AI | Honors robots.txt |
| meta-externalagent | Crawls for training and to improve Meta products | Honors robots.txt |
| Meta-ExternalFetcher | Fetches a page when a user's action triggers it | May bypass robots.txt |
Meta documents all three on its web-crawlers page for developers, with a robots.txt token for each. Verification is by network: Meta publishes the IP ranges its crawlers use through the routing registry for its autonomous system, AS32934, so a request carrying a Meta token from outside those ranges is not Meta. None of these is facebookexternalhit, the much older agent that fetches link previews when someone shares a URL on Facebook or Messenger. Blocking an AI agent does not touch previews.
What the indexer did on our site
16.3K reads in 30 days against a site of a few hundred pages means meta-webindexer returned to the same pages many times over. That is the sweep pattern: broad passes across the whole site, repeated, with revisits spread evenly rather than clustered around whatever people are asking about this week. It is the signature of an indexer keeping a copy fresh, not of a retrieval agent answering live questions. Our most-read pages by AI overall are the homepage, the State of the Bots report, robots.txt and the bot paywall page, and the indexer touched all of them repeatedly.
meta-externalagent's 2.3K has a different shape and a different purpose. Training reads are one-off value: a page ingested for training is worth something once. Index reads are recurring: the index is consulted every time a Meta AI user asks something your page can answer.
Neither sent a human back. In the same period we recorded 37 referrals from ChatGPT, 7 from Claude, 3 from Perplexity and none that identified as Meta AI. 16.3K reads, zero visible clicks. That ratio is the reason server logs, not referral reports, are where Meta shows up.
What Meta builds with it
Meta AI runs inside WhatsApp, Instagram, Facebook and Messenger, in the standalone Meta AI app and on Ray-Ban Meta glasses, across a family of apps Meta reports at over three billion daily users. In September 2026 Meta launched Muse, a personal assistant. Meta's documentation ties meta-webindexer to search inside Meta AI, so the index it builds is a distribution channel into the largest consumer messaging and social surfaces that exist. A page in that index can be the source of an answer in a WhatsApp thread with no link, no click and no trace in Google Analytics.
How to treat them
- Allow meta-webindexer. It is an AI search indexer. Disallow it only for content you never want in Meta AI's answers.
- Decide on meta-externalagent. It is a training crawl. Allow it, price it per fetch through a bot paywall, or disallow it with its token.
- Verify by network, not by string. The token is free to copy. The AS32934 ranges are not.
- Measure reads, not referrals. Meta sends no visible clicks, so the only evidence that Meta AI is reading you is in server logs. Bots do not run JavaScript, so nothing else sees them.
TollBit's Bot Index tracks both agents across 10,000+ sites, and the bot paywall sets per-agent terms, for example open for the indexer and priced for the training crawler.
Frequently asked questions
Is meta-webindexer new? It is newer than meta-externalagent, which Meta documented in 2024. On our site it is already the largest AI reader by a wide margin.
Will blocking Meta's AI crawlers break my Facebook link previews? No. Link previews come from facebookexternalhit, which is a separate agent with its own robots.txt token. A disallow for meta-externalagent or meta-webindexer leaves previews alone.
Why would an indexer read a small site 16,000 times in a month? Breadth times repetition. An indexer touches every page it can find and comes back to each one on a schedule that varies by platform, from hours to weeks. Multiply a few hundred pages by frequent revisits and 16K is an ordinary month.