Products
Solutions
Resources
Marketplace
About
Get started
Products
Licensed RAG access
Bot & agent paywall
Content controls
Agent Site
MCP
NLWeb
Portfolio analytics
TollBit Analytics
Solutions
Digital Publishers
Agentic commerce
Resources
Platform Docs
API & Integrations Docs
Scraper Index
Bot Index
State of the Bots Q2 2025
State of the Bots Q1 2025
State of the Bots Q4 2024
Marketplace
AI companies
About
Press
FAQ
Blog
Team
2024 Q4, The First Issue
Resource Center
Overview & platform docs
API Documentation
Agent Site white paper
State of the Bots Reports
2026 Q1 & Q2, The Bad Bots
2025 Q3 & Q4, The Leaky Pipes
2025 Q2, Good Net Citizens
2025 Q1, The Rise of RAG Bots
2024 Q4, The First Issue
Executive Summary
Foreword
Section 1: AI User Agent Landscape
Section 2: Scale of AI Scraping
Section 3: Referrals from AI
Section 4: Robots.txt Compliance
Appendix: AI User Agent Profiles
State of the Bots
2024 Q4, The First Issue