Markdown for AI Agents: Accept text/markdown, Content-Signal and Link Headers Explained

Short answer: Markdown for AI agents means that your server answers the same address with plain Markdown instead of HTML when a client asks for it with the request header Accept: text/markdown. Browsers keep getting HTML. It is a few lines of server code or a CDN feature you switch on, and it gives agents clean text without menus and scripts. A benefit for rankings or AI citations is not proven, so treat it as a cheap extra on top of readable HTML, a good robots.txt and a sitemap, never as a replacement.

Why do AI agents like Markdown?

A web page is built for browsers. Around the text you want to read sit navigation, a cookie banner, scripts, style classes and footers. A person's eyes skip all of that. A program that has to feed the page into a language model cannot skip it for free, because the model has a limited context window and every extra element uses part of it.

Markdown keeps what matters for reading: headings, paragraphs, lists, tables, links. That makes agents and similar tools natural candidates to ask for it, and it is why the llms.txt proposal is written in Markdown too. We have written about that file in What Is llms.txt?.

Cloudflare makes the size argument visible. In its announcement of Markdown for Agents, the example response contains an x-markdown-tokens header with an estimated token count (725 in the example), so a client can judge how much context a page will take.

What this does not tell you is whether the big AI search systems ask for Markdown. Cloudflare's announcement shows Radar data of Markdown responses to OAI-SearchBot. That proves such responses happen, not that the crawler requests them; OpenAI does not document it. So we do not claim it. What is documented is that some agents already ask for it, as the next sections show.

How content negotiation works: Accept, Vary and Content-Type

Content negotiation is an old part of HTTP. One address can have several representations, and the client says which one it prefers. Three headers carry the whole mechanism:

A browser sends text/html first and adds a wildcard */* as a fallback. An agent that wants Markdown names it explicitly, for example Accept: text/markdown or Accept: text/markdown, text/html;q=0.8. Cloudflare writes that common coding agents such as Claude Code and OpenCode already send such Accept headers.

The rule that keeps everybody safe is simple: answer with Markdown only if text/markdown is named explicitly and is not rated lower than text/html. The wildcard */* must never count. Otherwise a browser, a link checker or a social preview bot, all of which accept "anything", would receive Markdown.

Markdown for AI agents with Cloudflare

On 12 February 2026 Cloudflare announced Markdown for Agents. If it is enabled for your zone (it is in beta, at no cost, on the Pro, Business and Enterprise plans), a request with Accept: text/markdown receives your HTML converted to Markdown on the fly. Your origin server does not change.

The example response in the announcement shows the headers to expect:

content-type: text/markdown; charset=utf-8
vary: accept
x-markdown-tokens: 725
content-signal: ai-train=yes, search=yes, ai-input=yes

That is the quickest route if your site already runs behind Cloudflare on one of those plans and the converted result looks right. Check the result with curl (see "How to test it with curl") before you rely on it. If you want full control of what the Markdown version says, or you run on something else, build it yourself.

Build it yourself: a short Node example

The following sketch contains the rules from above: Markdown only for an explicit request, Vary: Accept on both versions, a fixed address /index.md for clients that do not negotiate, and a canonical link in the header of the Markdown answer. It is a minimal example, not production code.

import http from 'node:http';
import { readFileSync } from 'node:fs';

function wantsMarkdown(accept = '') {
  const weight = new Map();
  for (const part of accept.toLowerCase().split(',')) {
    const [range, ...params] = part.trim().split(';');
    const q = params.find((p) => p.trim().startsWith('q='));
    weight.set(range.trim(), q ? Number(q.trim().slice(2)) : 1);
  }
  const md = weight.get('text/markdown') ?? 0;
  const html = weight.get('text/html') ?? weight.get('text/*') ?? weight.get('*/*') ?? 0;
  return md > 0 && md >= html; // the wildcard alone never selects Markdown
}

http.createServer((req, res) => {
  if (req.url === '/index.md' || (req.url === '/' && wantsMarkdown(req.headers.accept))) {
    res.writeHead(200, {
      'Content-Type': 'text/markdown; charset=utf-8',
      'Vary': 'Accept',
      'Link': '<https://example.com/>; rel="canonical"',
    });
    return res.end(readFileSync('index.md'));
  }
  if (req.url === '/') {
    res.writeHead(200, { 'Content-Type': 'text/html; charset=utf-8', 'Vary': 'Accept' });
    return res.end(readFileSync('index.html'));
  }
  res.writeHead(404, { 'Content-Type': 'text/plain; charset=utf-8' }); // every other address answers 404
  res.end('Not found');
}).listen(3000);

Three details are worth a closer look:

• Vary on both versions. The HTML answer needs Vary: Accept as well, otherwise a shared cache can store the HTML and hand it to an agent that asked for Markdown, or the other way round.
• The fixed address. /index.md always returns Markdown, no matter what the client sends. Agents and people can link to it, and it works where headers cannot be set. Our own seogrinder.com/index.md does this.
• The canonical header. The Markdown version is a second representation of the same page. Google documents that rel="canonical" can be sent as an HTTP header for files that are not HTML. It is a strong signal that the Markdown version belongs to the page, not a guarantee (Google).

Write the Markdown file by hand or generate it from the same source as the HTML. Either way it should say the same as the page, only without the frame.

Link headers: pointing agents to the description of your site

The Link response header is defined in RFC 8288. It names a related address and the type of relation, for example Link: </llms.txt>; rel="describedby". The relation describedby is registered, and its meaning fits: the linked resource describes this one. A client that reads response headers learns where the description of your site lives before it parses any body.

seogrinder.com sends this header on its start page and points to its llms.txt. On the Markdown answer the same Link header also carries the canonical relation shown above, separated by a comma.

Whether any agent follows these links today is not proven. The cost is one line in the server configuration.

The Content-Signal line

In September 2025 Cloudflare introduced a line for robots.txt that says what a crawler may do with your content once it has fetched it:

Content-Signal: search=yes, ai-input=yes, ai-train=no

search covers indexing and snippets, ai-input means using the content directly in AI answers, and ai-train means training or fine-tuning a model. The Cloudflare announcement and contentsignals.org describe the format. It is a declaration, not a barrier: nobody is forced to respect it. Cloudflare says that content signals express preferences, that they are not technical countermeasures and that some companies might simply ignore them. As shown above, Cloudflare's Markdown example also carries a content-signal response header.

The line belongs in your robots.txt, with all the other rules for AI bots. The details, including the common mistake with separate groups per bot, are in robots.txt for AI Crawlers. The robots.txt of seogrinder.com uses one shared group for all crawlers and nine AI identifiers, with Content-Signal: search=yes, ai-input=yes, ai-train=yes.

How to test it with curl

Check with the same tool an agent would use. The -I option shows only the headers:

curl -sI -H "Accept: text/markdown" https://example.com/

Look for four things in the answer: status 200, content-type: text/markdown, vary containing Accept, and a link header with the canonical address. This is the result for seogrinder.com, shortened to the relevant lines:

HTTP/2 200
content-type: text/markdown; charset=utf-8
vary: Accept
link: </llms.txt>; rel="describedby"; type="text/plain", <https://seogrinder.com/>; rel="canonical"

Then run the opposite checks. Each of these must return text/html:

curl -sI -H "Accept: text/html" https://example.com/
curl -sI -H "Accept: */*" https://example.com/
curl -sI https://example.com/

Last, send -H "Accept: text/markdown, text/html;q=0.8". Markdown should win, because it is rated higher.

How SEOgrinder checks it

Open the AI readiness tab in SEOgrinder after a scan. Under "Checks for AI agents" you find seven lines. Three of them concern this article, and all three sit in the group "Extras":

• Serve text instead of HTML (Markdown Negotiation): met when a request with Accept: text/markdown returns Markdown.
• Guide agents to your data (Link Headers): looks for useful relations in the response header.
• Declare how the content may be used (Content Signals): looks for a complete Content-Signal line that covers ai-train, search and ai-input.

Every line has the button Copy agent prompt, and a met line also expands to show the text it found. For an open point, the prompt describes what to create, shows the current state with curl and gives a command to check the result. For a met point, it asks your coding assistant to review what exists, change only what is necessary and ask you before removing a rule it does not understand. Decisions such as allowing or blocking AI crawlers stay with you, and some open-point prompts suggest example values, so decide first and read the prompt before you paste it into Claude Code, Cursor, GitHub Copilot or ChatGPT. Then press Check again to rescan. Reports are cached for up to 24 hours, and Check again forces a fresh run (a few times per hour).

SEOgrinder is honest about the group "Extras": these are suggestions from individual vendors, and an effect on Google is not proven. The tool also does not measure whether ChatGPT, Gemini or Perplexity actually mention your brand. It checks the technical base: can machines reach and read your page?

Is it worth the effort?

It depends on what your site offers. Documentation, reference pages, long guides and knowledge bases are the kind of content that agents fetch to answer a concrete question, and a clean Markdown version is useful there. For a small shop or a local business with five pages, the time is better spent on the basics: text that is present in the HTML without JavaScript, a robots.txt that lets the search bots in, and a sitemap.

For Google's own AI features, Google says that there are no additional requirements to appear in AI Overviews or AI Mode nor other special optimizations necessary; the details are in Google's documentation. A Markdown version is therefore not a requirement there. Other assistants may behave differently, but nobody can promise that an AI assistant will name your page.

Common mistakes

• No Vary header. Without Vary: Accept, a cache or CDN can serve the wrong version to the next visitor.
• A cache in front that ignores the header. Test through the public address, not only against your origin server.
• Markdown for browsers. If your check treats the wildcard */* as a request for Markdown, visitors see raw text. Test with a normal browser.
• No canonical link. The Markdown version can then be treated as a separate page. We chose the canonical header instead of noindex because the negotiated version lives under the same address as the page.
• A Markdown version that says something else. Keep the content the same as the HTML page, only without the frame. We have no source on how search engines treat differences between the two, so keep them identical.
• An outdated file. A hand-written index.md that lists last year's offer misleads agents. Generate it from the source of the page, or put it on your checklist.
• An empty HTML page. If the HTML itself has no text until JavaScript has run, Markdown for agents does not fix the other clients, such as social previews. SEOgrinder flags pages that are only assembled in the browser.

FAQ

Is Markdown for AI agents a ranking factor?

There is no evidence for it. Treat it as a delivery format for machines, not as a ranking lever.

Do ChatGPT, Claude or Perplexity request Markdown?

Cloudflare writes that coding agents such as Claude Code and OpenCode already send Accept: text/markdown. For the search crawlers of the large vendors there is no documentation that they ask for it; Cloudflare's Radar data shows Markdown responses to OAI-SearchBot, which is not proof of a request header. So it is not proven.

Do I need Cloudflare?

No. Cloudflare offers the conversion as a feature (in beta, on the Pro, Business and Enterprise plans). The Node sketch above shows that a few lines of server code are enough, and the same rules can be built into any server or framework.

Should I use content negotiation or a separate .md address?

You can do both. A fixed address such as /index.md is easy to link and to test. Negotiation keeps one address for both audiences. seogrinder.com offers both for its start page.

Will normal visitors ever see the Markdown version?

Not if you follow the rule about the wildcard: Markdown is sent only when text/markdown is named explicitly and not rated lower than text/html.

Your next step

Run the free scan in SEOgrinder and open the AI readiness tab. Look at the three Extras lines for Markdown, Link Headers and Content Signals, and decide whether your site is the kind that agents read. If it is, copy the agent prompt, let your assistant make the change, check the result with curl, and rescan.