Markdown vs HTML for AI agents: which should you serve?
Serve Markdown to bots that fetch a document through an API, and serve slim, server-rendered HTML to bots that fetch your pages at their URLs. Markdown wins on token cost. Semantic HTML wins on structure, links and the ability to act. The wrong answer is the one most sites give today: a full browser page to a visitor that reads only text.
Why page weight matters to an AI visitor
A language model does not see a page. It reads tokens, and every tag, script and tracking pixel in your HTML is a token it pays for and spends attention on. For retrieval (RAG) systems the context window is working memory. Fill it with boilerplate and there is less room for the facts, the answer gets worse, and the fetch costs more.
Measured across ten publisher articles on our network, the HTML a browser downloads versus the Markdown an agent needs:
- A Forbes article: 1,201 KB of HTML, 41 KB as Markdown, 97% smaller.
- A Goal.com article: 378 KB to 13.2 KB.
- Across all ten, 56% to 97.5% smaller.
RAG cannot cache the internet, so those fetches repeat. The weight you remove is removed on every read, and from your CDN bill.
Where Markdown is the right format
Markdown keeps headings, lists, links and emphasis and drops presentation. It is the natural format when an agent asks for a document rather than a page.
- Fetch APIs and licensed RAG. When a developer's pipeline requests content by URL and pays per fetch, it wants the article as text with licensing metadata, not a rendering. We have returned Markdown on this path since our first day.
- Coding agents and research agents. On tollbit.com the
claude-codeuser agent fetched our State of the Bots report about 1,350 times in 30 days. A developer piping a report into a context window wants Markdown or a table, not a slide deck. - Long documents. Reports, documentation, manuals. Structure is flat, links are few, text is the point.
Markdown's limits are real. Multi-axis tables lose nuance and there is no form to submit.
Where slim HTML is the right format
Retrieval bots overwhelmingly fetch ordinary HTML at ordinary URLs. ChatGPT-User, Claude-User and Perplexity-User request the page a person would open, and they request it in HTML. For them the goal is not a different file type but a lighter page.
Slim semantic HTML means the page the agent receives at the same URL, server-rendered, with the article, product data and links intact, and without the overlay scripts, ad slots, chat widgets and client-side rendering. It keeps three things Markdown weakens:
- Links with context. Navigation an agent can follow to the next page, which is how retrieval bots discover pages.
- Tables and structured data. Price, stock, specification, in markup the agent can parse reliably.
- Actions. Forms, buttons and checkout. Computer-use agents complete tasks on HTML. In the Agent Sites benchmark with KERNEL, a cleaned HTML surface alone gave 24 to 35% faster task completion and up to 38% fewer steps.
Side by side
| Markdown | Slim semantic HTML | |
|---|---|---|
| Best for | Documents fetched through an API | Pages fetched at their URL |
| Token cost | Lowest | Low, if scripts and overlays are removed |
| Links and navigation | Present, but flat | Full, follows site structure |
| Forms and actions | None | Yes, for computer-use agents |
| Who asks for it | RAG pipelines, coding agents, licensed fetch | Retrieval bots, search indexers, task agents |
What this means for your site
You do not choose once. The agent-facing version of your site can return Markdown to a fetch API and slim HTML to a bot hitting a URL, from the same origin, with the same facts as the human page. That is what an agent site does: one rail on your own domain, with the format decided per visitor.
Frequently asked questions
Should I publish a Markdown mirror of every page? Not as a separate site. Retrieval bots fetch your real URLs, so a parallel mirror sees little traffic. Serve lighter responses at the URLs agents already fetch, and offer Markdown where an API fetch asks for it.
Is slim HTML the same as the AMP pages we used to build? Similar idea, different visitor. AMP was a lighter page for mobile humans within Google's rules. Slim HTML for agents is a lighter page for bots, under your rules, and Googlebot never sees it.
Does any of this change what humans see? No. Humans and search engines keep the full page. Only identified AI agents are routed to the lighter version, and the facts are the same on both.