2025, February 24
AI Scraping Is On The Rise.
TollBit State of the Bots - Q4 2024
The TollBit AI User Agent Index is our first quarterly report that tracks overall trends in the scraping habits of the AI industry. Through our unique position, we have created this comprehensive index across our publisher network to give a never before seen picture of scraping activity and trends across the industry.
The pace of scraping growth accelerated; scrapes per website doubled from Q3 to Q4 and scrapes per page more than tripled.
AI bot scrapes that bypassed robots.txt grew by over 40% between Q3 to Q4. Blocking AI bots via robots.txt remains an insufficient mechanism to prevent unwanted scraping.
TollBit sees that AI chat bots on average drive referral traffic at a rate that is 96% lower than traditional Google search.
Even when blocked, apps like Perplexity continue to access sites through unidentified user agents. Unidentified user agent scraping was roughly equivalent to identified AI bot scraping Q4.
