How do AI agents discover new pages on a website?
Three ways: AI search indexers sweep your robots.txt and sitemap on a periodic schedule, retrieval bots follow links on the pages they were sent to, and every class of agent reaches you through third-party pages that link to you. There is no submission form. A new page is discovered when one of those three paths reaches it, and how soon that happens varies by platform, from hours to weeks.
Path 1: indexers read robots.txt first, then the sitemap
AI search indexers (OAI-SearchBot for ChatGPT search, Claude-SearchBot, PerplexityBot, Meta's meta-webindexer) build the index an assistant searches before it fetches anything live. They behave like a shallow, periodic version of a search engine crawler.
Our own logs show the pattern plainly. On tollbit.com, Sep 15 to Oct 14 2026, OAI-SearchBot's single most-fetched URL was robots.txt, about 1,000 fetches, ahead of the homepage. Claude-User's most-fetched URL was also robots.txt, nearly 600 fetches. The robots.txt file is where the sitemap is declared, so an indexer that reads it that often is checking what it is allowed to fetch and where the list of URLs lives. meta-webindexer, the largest single reader of our site at 16.3K successful fetches, sweeps broadly from that list.
For a new page, this means the fastest discovery path is: list it in the sitemap the bots read, and keep robots.txt pointing at that sitemap.
Path 2: retrieval bots follow links from the page they landed on
Retrieval bots (ChatGPT-User, Claude-User, Perplexity-User and their peers) fetch pages live because a person asked a question. The assistant usually arrives with one URL from a search step. If that page does not fully answer the question, the agent follows links on it. In our logs ChatGPT-User, with 10.5K successful reads, moves across product pages: the homepage, publisher solutions, licensed RAG, the bot paywall page. It reads the way a person skims a site, one link at a time.
A new page that is linked from the pages agents already fetch gets discovered the next time an agent needs what it says. A new page linked from nowhere, in no sitemap, is not discovered by retrieval at all.
Path 3: third-party pages that are fetched often
Agents read far more third-party coverage than brand sites. Round-ups, comparisons, directories and trade coverage are fetched daily because they answer the comparison questions people ask. A link to your page from one of those surfaces gets followed. This is how most brands first appear in agent reads, before any agent has fetched the brand's own site.
You cannot control those pages, but you can see which of them carry your category, and that list is a better target than instinct. TollBit Intelligence reads across 10,000+ sites and shows where a topic's reads concentrate.
How long discovery takes
Recrawl schedules differ by platform and by bot class, from hours to weeks. Indexers revisit on their own cadence. Retrieval bots fetch when asked, so a page on a topic people ask about gets fetched sooner than a page nobody asks for. AI apps cannot cache the internet, because RAG loads pages fully into a context window, so pages that matter get re-fetched repeatedly once they are known.
The practical consequence: discovery is fast for pages that answer live questions and slow for pages that do not. Writing the question into the title and the answer into the first two sentences shortens the wait.
What a bot-only sitemap changes
When TollBit runs the agent-facing version of your site, it serves the robots.txt and sitemap that AI agents read. That sitemap can differ from the one Google reads. It can list agent-only pages that are not on the human site, drop expired URLs that waste agent crawl budget, and carry per-platform versions. Google and Bing are never routed to the agent origin, so their sitemap and their index are untouched.
Discovery then stops being a hope and becomes a check: the page is listed, the indexer fetches the list, and the first fetch of the new URL shows up in server logs by agent and by date.
Frequently asked questions
Should I submit agent-only pages to Google Search Console? No. Agent-only pages live only in the sitemap served to AI agents. Search engines read your human sitemap and your human pages, which do not change.
Does an llms.txt file help agents discover pages? In observed network logs, alternative bot-facing files see marginal fetch volume compared with robots.txt and ordinary HTML pages. It does no harm, but the sitemap in robots.txt is what indexers actually read.
How do I know a new page has been discovered? Server logs record the first fetch by each agent user agent, with a date. Google Analytics will not show it, because bots do not run JavaScript. A server-side AI analytics view lists new URLs as they are first read, per agent.