OAI-SearchBot and GPTBot: what they are and what they read
OAI-SearchBot builds the index behind ChatGPT search. GPTBot collects pages that may be used to train OpenAI's models. Neither fetches a page because a person asked about it, which is the job of the third OpenAI agent, ChatGPT-User. On tollbit.com between September 15 and October 14 2026, OAI-SearchBot made about 1.4K successful reads, with robots.txt its single most fetched URL at about 1,000 fetches, and GPTBot made about 0.7K.
How they declare themselves
Both carry a Mozilla-compatible prefix, a named token with a version, and a link to OpenAI's documentation.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.x; +https://openai.com/gptbot
OpenAI publishes the IP ranges for each: https://openai.com/searchbot.json for OAI-SearchBot and https://openai.com/gptbot.json for GPTBot. Verify requests against those lists. The robots.txt tokens are OAI-SearchBot and GPTBot, and OpenAI's documentation states that both honour robots.txt directives.
The three OpenAI agents side by side
| GPTBot | OAI-SearchBot | ChatGPT-User | |
|---|---|---|---|
| Class | Training bot | AI search indexer | Retrieval (RAG) bot |
| Trigger | Scheduled bulk crawl | Scheduled sweep, sitemap-driven | A person's question |
| Effect of a robots.txt disallow | Pages left out of training collection | Pages left out of ChatGPT search results | Live fetches may continue |
| Reads on tollbit.com, 30 days | 0.7K | 1.4K | 10.5K |
| Most fetched URL here | Spread across the site | robots.txt, about 1,000 times | Homepage, then product pages |
The lesson in the last two rows is the shape of OpenAI traffic on a typical site. Training is the smallest visitor. Indexing is steady and checks the rules before every sweep. Retrieval is the big one, because people ask questions far more often than indexes are rebuilt or models are trained.
What OAI-SearchBot did on tollbit.com
About 1,000 of its 1.4K fetches were robots.txt. The next most fetched URLs were the homepage and the careers page. That is indexer behaviour: read the rules, read the entry points, follow the sitemap, repeat. The careers page is a reminder that an indexer sweeps what is linked and listed, not what you would like it to prioritise.
Two consequences for a site owner. First, the sitemap OAI-SearchBot reads is the lever. If you want a page in ChatGPT search, it should be in the sitemap served to bots, and expired URLs should be out of it. Second, a bot that fetches robots.txt a thousand times a month is one that reads your access rules constantly.
What GPTBot did on tollbit.com
Its 0.7K reads were spread across the site with no single dominant URL, which is what bulk collection looks like. GPTBot is the agent most publishers mean when they say they have blocked OpenAI. It is also the one whose absence changes answers least, since model training runs on a slow cycle and live answers come from retrieval.
How to treat each
- GPTBot. Decide on training terms. Disallow it if you do not want pages in training collection. Publishers on the TollBit network can instead price it: an HTTP 402 response with a per-fetch rate, so collection pays rather than stops.
- OAI-SearchBot. Allow it if you want to appear in ChatGPT search. Serve it a bot-only sitemap that lists the pages you want indexed, including agent-only pages, and omits dead URLs.
- ChatGPT-User. Serve it readable, current text. It is the agent that puts your facts into the answers people read.
All three execute no JavaScript, so none of them appears in Google Analytics. Count them in server logs, by agent and by page, and segment chatgpt.com as a referrer for the human clicks that come back. On tollbit.com that was 37 referrals against 12.6K OpenAI reads in the period.
Frequently asked questions
If I block GPTBot, am I removed from ChatGPT? No. GPTBot governs training collection. ChatGPT search uses OAI-SearchBot's index and live answers use ChatGPT-User. Each has its own token, so block or allow them separately.
Why does OAI-SearchBot fetch robots.txt so often? An indexer re-reads the rules before each sweep across many paths and time windows. On our site that was about 1,000 fetches of a file that changed rarely. It is a sign of compliance, not a problem.
How do I know a GPTBot request is really from OpenAI?
Check the source IP against openai.com/gptbot.json. User agent strings are trivially spoofed. Across our network 12.7% of traffic claiming to be another vendor's retrieval agent was spoofed in H1 2026.