NEWTollBit State of the AI Bots - Q2 2025
Content controls

Control what AI systems can access from your site

Maintain full control over which elements, pages, and formats are included or excluded when AI developers request your content

Problem

When AI systems access your content, they don't distinguish between what you want shared and what you don't. Images, videos, embedded widgets, syndicated content, and sensitive material all get scraped the same way.

Publishers need a way to:

  • Exclude specific elements like images, videos, and third-party embeds
  • Protect syndicated or editorially restricted content from AI access
  • Return clean, structured content without changing the live site
  • Apply different rules for different AI use cases

Benefits

Explicit control, not assumptions

Choose exactly what AI systems can and cannot receive

Protect sensitive and third-party content

Exclude entire pages or article types from AI access

Cleaner inputs for AI use cases

Return structured, machine-readable content

No impact on your live site

Controls apply only to AI requests routed through TollBit

Solution

Step 1

Route AI requests through TollBit

When AI developers request content via TollBit, content controls are applied before anything is returned

Step 2

Apply element filters

Select which HTML elements should be excluded from AI responses, including:
  • Images
  • Videos
  • Audio
  • iFrames
  • Links
These filters remove entire element types from the returned content while leaving the original page untouched.

Step 3

Use advanced class-based rules

For more precise control, exclude elements by CSS class. This allows you to remove:
  • Embedded media players
  • Sponsored or third-party widgets
If it is styled consistently on your site, it can be filtered.

Step 4

Apply article filters

Exclude entire pages from AI access based on HTML tags and attributes. Common examples include:
  • Syndicated articles
  • Editorially restricted content
If a page matches an article filter, it is not accessible to AI systems at all.

Step 5

Format approved content for AI

Only allowed content is cleaned and structured for AI use

Built for the ways AI actually interacts with the web

TollBit supports multiple AI use cases with purpose built solutions that are tailored to how agents interact with content and commerce

Digital publishers

Monitor, control, and monetize AI access. Publishers face growing AI traffic that is difficult to track and even more difficult to control. TollBit gives publishers a free platform for understanding how agents interact with their sites, separating AI traffic from human traffic, and enforcing rules and rates for content access.

Agentic commerce

Enable reliable, measurable agent-driven actions. Commerce platforms use TollBit to separate agent access from the human site, require agent identity, expose machine-facing workflows, measure real usage, and attach economics to agent activity, without disrupting existing commerce infrastructure.

Trusted by world's best publishers

Protect sensitive and restricted content

Ensure AI systems only access the parts of your site you explicitly allow
TollBit