Are AI companies paying for content? Pay-per-crawl explained
Yes, through three mechanisms that arrived in order. First came bulk licensing deals between AI companies and the largest rights-holders. Then court settlements began putting prices on past training use. The third, now operating at scale, is per-fetch payment: a price on each bot request over HTTP, instead of a negotiated archive. In H1 2026, 2.6B+ AI bots were directed to the TollBit Bot Paywall. That last mechanism is the only one that reaches everyone below the top of the market.
The three mechanisms
1. Bulk licensing deals. Multi-year, negotiated, headline-grade. News organisations and wire services first, then in 2026 the music industry, where AI music companies signed licences with major rights-holders under litigation pressure. Reported values run from tens to hundreds of millions of dollars. They work for parties large enough to negotiate them.
2. Settlements and rulings. Courts are pricing the past. A widely reported 2026 settlement over books used in AI training reached into the billions. European rulings against AI music generation established that training on protected catalogues without a licence carries liability. Every settlement resets the floor for the next deal.
3. Per-fetch payment (pay-per-crawl). The web-native mechanism: a price per request rather than a contract per publisher. This is where the long tail of the web, every site that will never get a bulk deal, participates.
How pay-per-crawl works
The mechanics run over standard HTTP.
- A bot requests a page. The server identifies it as an AI agent (user agent, IP verification, request signature).
- Instead of the page, the server returns HTTP 402 Payment Required with the site's rate for that content.
- The bot's operator pays per fetch through the payment rail and receives the content, typically in a bot-optimised format.
- Every transaction is logged: who fetched what, when, at what price.
Who pays, and how rates are set
The operator of the agent pays: the company running the assistant, the search product or the research agent. The site owner sets the rate, and sets it per bot class, per path and per freshness, so a training crawler, a retrieval bot and a search indexer can each see a different price or no price at all. Across our network of 10,000+ sites, rates vary by orders of magnitude by content type. There is no single market price, by design.
A licensing deal versus a toll
| Bulk licensing deal | Per-fetch toll | |
|---|---|---|
| Who gets one | Dozens of large rights-holders | Any site that can identify a bot |
| What is sold | An archive, often for training | Each live fetch, mostly for retrieval |
| Pricing | Negotiated, fixed term | Set by the owner, changed any time |
| Revenue shape | One-off or annual | Recurring, scales with AI usage |
| Record | The contract | A log of every fetch |
| Time to start | Months | Days |
They coexist. A deal suits the largest archives and closest partnerships. A toll covers live retrieval, and everyone the deal market never reaches.
Why per-fetch is the mechanism that generalises
- Bulk deals cover dozens of publishers. The web has millions. A price per request needs no negotiation to include a new site.
- Retrieval is recurring. Assistants fetch pages at answer time, continuously, because RAG cannot cache the internet. Fresh content is fetched most, so value tracks ongoing publishing rather than back catalogue alone. In the EU, AI scrapes outnumbered AI referrals 227 to 1 in Q2 2026. The fetch is the event that scales.
- It aligns both sides. AI operators get legitimate, reliable, machine-readable access instead of adversarial scraping. Publishers get payment and a record of exactly what was taken.
Where TollBit fits
TollBit has run a bot paywall and licensed RAG rail across 10,000+ publisher sites for about three years, on top of the CDN or bot management a site already uses. Publishers named in press as customers include the Associated Press, Gannett and TIME.
Frequently asked questions
How much does one fetch cost? It depends on the site. Owners set rates by content type, section and freshness, and across the network they range across orders of magnitude. A news page published this morning and an archive page from 2015 do not carry the same price.
Is pay-per-crawl one company's product? No. Several implementations exist, including per-fetch approaches from CDN providers for their own customers. TollBit operates one of the largest, CDN-agnostic, across 10,000+ sites, with per-agent rate cards and a licensed fetch API for developers.
Does paying for fetches replace licensing deals? They coexist. Bulk deals suit the largest archives and closest partnerships. Per-fetch pricing covers live retrieval and everyone the deal market never reaches, and a publisher with a deal can still toll the agents outside it.