2024 Q4, The First Issue

Resource Center

Overview & platform docsAPI DocumentationAgent Site white paperState of the Bots Reports

2026 Q1 & Q2, The Bad Bots2025 Q3 & Q4, The Leaky Pipes2025 Q2, Good Net Citizens2025 Q1, The Rise of RAG Bots2024 Q4, The First Issue

Executive Summary

Foreword

Section 1: AI User Agent Landscape

Section 2: Scale of AI Scraping

Section 3: Referrals from AI

Section 4: Robots.txt Compliance

Appendix: AI User Agent Profiles

State of the Bots

2024 Q4, The First Issue