Is my website readable by AI agents?
You can find out in ten minutes with a terminal and a text editor. Fetch your most important pages the way a bot does, with no JavaScript and an AI user agent, and count the words that come back. If the answer is a few dozen words of navigation and a loading message, your site is not readable, whatever it looks like in a browser.
Readable means a simple bot or crawler can fetch the page, receive the content as text, and find the answer without executing anything. Most sites built in the last ten years fail at least one of the checks below on at least one important page.
Check 1: fetch as a bot and count the words
Use curl or any HTTP client. Request the page with the user agent of a real retrieval bot, for example ChatGPT-User or Claude-User, and save the response. Then strip the tags and count what is left.
What you are looking for:
- Readable words. A product page should return the product name, price, description, specifications and availability as text. If it returns under 100 words, the content is being added by JavaScript after load, and a bot never sees it.
- The answer near the top. Bots read the whole page but weight the beginning. A page that opens with 400 words of menu labels and cookie notice buries the answer.
- The status code. 200 is a page. 403 is a block, often from a bot manager's default rule. 404 is a page that does not exist. 429 and 5xx are a site that cannot keep up. Every non-200 is an answer written without you.
Do this for the pages that matter: the homepage, the top ten products or services, pricing, the FAQ, and any page that answers a question like "is X safe" or "does X work with Y".
Check 2: JavaScript-rendered pages
Single-page applications, headless front ends and client-side product configurators share a failure mode. The HTML the server sends is a shell. The content arrives when a script runs. Bots do not run it. A human sees a complete page, a bot sees the shell, and neither notices they are looking at different things.
The fix is server rendering for the pages bots fetch, or a rendering served to bots only. Either way the test is the same: the first response must contain the words.
Check 3: PDFs, slides and images of text
Spec sheets as PDF, pricing as a screenshot, a comparison chart as an image, product manuals as embedded slide decks. Some bots open a PDF, many do not, and a crawler does not read text out of a PNG. If the only place a fact lives is inside a file or a picture, that fact is not readable. Put the facts in HTML too. A table of specifications beats a downloadable datasheet every time.
Check 4: the sitemap
Open your sitemap.xml with the same bot user agent. Count the URLs. Then sample twenty and fetch each one. A healthy sitemap returns 200 for everything in it. An unhealthy one lists retired products, redirected URLs and staging leftovers, and an AI search indexer sweeping it spends its crawl budget on dead ends. On our own site, OAI-SearchBot's single most-fetched URL over the past month was robots.txt, about 1,000 times, and Claude-User's was also robots.txt, nearly 600 times. Indexers and retrieval bots check the map before they read the territory.
Check 5: what the logs say
The checks above are a snapshot. Your server logs are the record. Filter for AI user agents (GPTBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Amazonbot, meta-webindexer and the rest) and look at three things: how many fetches returned non-200, which URLs they requested that do not exist, and which pages they read most. Google Analytics cannot show you any of this, because bots do not execute the script that reports to it.
One caution when reading unreachable URLs. On tollbit.com most of the "missing" pages agents ask for are vulnerability-scanner probes (/.git/config, /wp-admin) sent under AI-looking user agents. Those are not content gaps. The real gaps are product and help URLs that an agent guessed and you never built.
What "zero readable words" looks like
The pattern is always the same. A brand site with thousands of products, each page rendering cleanly in Chrome. Fetched as a bot, every product page returns the same few dozen words: the brand name, the menu labels, a loading message, a cookie notice and a copyright line. Thousands of pages, one sentence of content, and every AI answer about those products assembled from third-party sites instead.
Where TollBit fits
An agent site solves this without changing the human site. TollBit fetches your page, renders it server-side as text or slim HTML, and serves that to identified AI agents on your domain, with a bot-only sitemap and server-log analytics showing which agents read which pages and whether they succeeded.
Frequently asked questions
My pages pass Google's mobile-friendly test. Does that mean they are readable by AI? No. Googlebot renders JavaScript. Most AI agents do not. A page can be fully indexed by Google and return a near-empty shell to ChatGPT-User.
Do I need to make the whole site readable? Start with the pages that answer buyer questions: products, pricing, specifications, FAQs, comparisons. Those are the pages retrieval bots fetch most.
How often should I re-run the checks? After every front-end release, and monthly against the logs. A redesign that moves content into a client-side component can turn a readable site into an unreadable one overnight.