Markdown vs HTML for AI: I Publish Both. Here Is What Actually Gets Read

Every page on this site is published twice. The version you are reading is HTML. The other is a plain markdown file at this same address with .md added to the end, built for AI crawlers and agents.

So when the markdown vs HTML for AI argument comes up, I do not have to guess. I run both formats in production, and I checked the actual numbers before writing this.

Here is the honest split: markdown wins on weight by a factor of about 20, and it still earns you nothing in AI search on its own. Both halves of that sentence matter.

Markdown vs HTML: what is the actual difference?

Markdown is plain text with a few symbols: # for headings, * for lists, pipes for tables. HTML is the full language of web pages: tags, attributes, scripts, styles, metadata. The same article can exist in both; the difference is how much machinery wraps the words.

The reason AI people care about the html vs markdown debate is simple: language models are soaked in markdown. Microsoft maintains MarkItDown, a converter with over 170,000 GitHub stars whose whole job is turning PDFs, Word files and HTML into markdown, because as its README puts it, mainstream LLMs natively "speak" markdown. Jina Reader does the same for any URL before an LLM reads it.

Notice the direction of every one of those tools. The industry converts to markdown at the point of reading. That part of the argument is settled.

How much smaller is markdown than HTML? My real numbers

I pulled three of my own posts twice each, once as the HTML page and once as the markdown twin, straight off the live server.

By the rough four-characters-per-token rule, that first post is about 118,000 tokens as raw HTML and about 6,200 as markdown. Same words, same tables, same links.

A model that eats the raw HTML pays a 20x token tax for content it did not ask for: navigation, scripts, styles, JSON-LD, hydration payloads.

One honest nuance before you rebuild your site over this. Serious AI pipelines do not feed raw HTML to the model; they run extractors that strip the noise first. So the real question is not whether markdown is lighter. It is whether serving markdown changes what AI systems actually do.

Do AI crawlers actually fetch markdown pages?

Mostly no. This is where the popular advice falls apart, and I am saying that as someone who serves markdown on every page.

Otterly ran the clean version of the experiment: .md copies of live pages published beside the HTML, equal discoverability, 14 days of watching. The .md URLs got zero AI crawler visits and zero citations. Every citation the test pages earned went to the HTML version.

Profound ran a bigger one across 381 pages on six websites for 21 days. The markdown group earned roughly one extra median bot visit over three weeks, and the difference was not statistically significant. Their own conclusion: no slam dunk.

A third test from SEO Depths served markdown directly to OpenAI's bots and measured delivery: content download dropped to about 40ms from 169ms, and the author still landed on the same verdict. The win is control and cleaner output, not rankings.

AI search engines read your HTML and clean it themselves. They do not go looking for a markdown menu.

Why do I still serve markdown twins, then?

Because search crawlers are not the only machines reading the web now. Agents are the other audience, and they behave differently.

When a person points a coding agent or an AI assistant at your site, that agent can fetch the .md twin and get the whole article inside a context budget where the raw page would not fit. A 6,200-token file is a polite guest; a 118,000-token one is not. This is the same logic behind llms.txt, which AI search engines also do not read, and which I generate anyway for exactly this audience.

The other reason is cost. My twins are generated at build time from the same source files as the pages. Zero extra writing, zero maintenance, no way for the two versions to drift apart.

And the structure survives. Headings, tables and links, the things engines actually quote, arrive clean instead of after a lossy extraction.

How do you serve markdown for AI crawlers properly?

If you do it, do it this way. Each of these choices got tested on this site, one of them painfully.

1. Give the twin its own URL. /insights/post gets a sibling at /insights/post.md. Generate it at build time from the same source.
2. Advertise it in the head: <link rel="alternate" type="text/markdown" href="/insights/post.md">, plus an HTTP Link header if you can.
3. Noindex the twin. Serve it with X-Robots-Tag: noindex so Google never treats it as a duplicate of the real page.
4. Unfold your components. If your pages use shortcodes or MDX blocks, the twin must render them out to plain markdown. Stripping them blindly deletes the most quotable parts of the page.
5. Do not content-negotiate. I first served markdown from the page's own URL when a client sent Accept: text/markdown. Cloudflare normalised that header away before it reached my origin, so the feature was silently dead. Worse, with edge caching on, one markdown response landing in the cache could have served raw markdown to every browser on that URL. Separate .md URLs, no negotiation.

You can check a working setup in one line:

curl -sI https://theguidex.com/insights/llms-txt.md
# content-type: text/markdown; charset=utf-8
# x-robots-tag: noindex
# x-markdown-tokens: 6202

That last header is a build-time token estimate stamped on every twin, so an agent can budget its context before it commits to the fetch.

Does markdown replace HTML for SEO?

No, and do not let anyone talk you into trying.

Your HTML page is where the canonical URL, the Open Graph tags, the schema, the byline and the internal links live. Those are the signals engines use to work out what a page is and whether to trust it, and a bare .md file carries none of them. Otterly's test pages showed the result: engines cited the HTML precisely because it is the version wired into the web.

The visible HTML body is also what gets extracted, chunked and quoted in AI answers. If you want the citation, the work is making your visible content worth quoting, and letting the right bots in, which is a robots.txt job, not a file-format job.

Markdown is a copy for machines that already found you. HTML is how you get found.

Final take

Markdown vs HTML for AI is a false fight. They are not competing for the same job.

HTML is the page: indexed, ranked, extracted, cited. Markdown is the takeaway copy: light, clean, and read by agents that a human pointed at you. On this site the twin costs nothing to produce, so I serve it and expect nothing from it in search.

If your build can generate markdown twins for free, do it. If someone is selling markdown as an AI search tactic, ask them for the citation data. I looked, and it only ever points one way.