2025, September 8
AI Scraping Is On The Rise.
TollBit State of the Bots - Q2 2025
The next AI visitors won't look like bots
Today's AI browsers and devtools, from Perplexity Comet to Firecrawl to Browserless, look human in site logs— showing up as Chrome while loading pages, ads, and solving CAPTCHAs.
Human traffic trended downward as AI traffic volume continued to increase. This quarter, human visits decreased 9.4% across all sites on TollBit.
AI visits rise as human visits fall. At the start of Q1, 1 out of every 200 visitors was AI. Now it is 1 in every 50, reflecting a 4x increase in the relative volume of AI visitors.
AI bots are continuing to ignore robots.txt directives this year. In Q2, 13.3% of AI bot visits bypassed robots.txt directives.
AI bot redirects to the TollBit Bot/Agent Paywall, have increased 360% from Q1 to Q2 2025 as websites have increasingly taken active measures to charge AI traffic.
