2025, June 11
AI Scraping Is On The Rise.
TollBit State of the Bots - Q1 2025
This TollBit State of the Bots report is our second quarterly report that tracks overall trends in the scraping habits of the AI industry. Through our unique position, we have created this comprehensive report across our publisher network to give an evolving view into how AI companies access content, revealing emerging patterns, behaviors, and shifts each quarter.
Retrieval-Augmented Generation (RAG) bots scrapes now exceed Training bot scrapes across the TollBit network. RAG bot scrapes grew 49% from Q4 to Q1.
Among websites that had Analytics set up before January 2025, traffic volume nearly doubled in Q1, rising by 87% during the quarter.
Despite more publishers trying to block bots, over 26 million AI scrapes bypassed robots.txt in March alone. The share of bots ignoring these files jumped from 3.3% to 12.9% in just one quarter.
As AI scraping increases, publishers are exploring more proactive defenses that don't depend solely on the honor system of robots.txt. Bot traffic directed to the TollBit Bot Paywall has increased by 732% from Q4 2024 to Q1 2025.
