robots.txt for AI Crawlers: Allow or Block GPTBot, ClaudeBot and PerplexityBot

By SEOgrinder · Published November 2, 2026 · Updated November 4, 2026 · 7 min read

Short answer: A robots.txt for AI crawlers decides, bot by bot, whether an AI system may read your site. There are three kinds of bots: training bots (GPTBot, ClaudeBot), search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and bots that fetch a page because a user asked (ChatGPT-User, Claude-User). If you want to be citable in AI search answers, allow the search bots. Whether you allow training is your decision. One small file does all of this, but one small mistake in how it is built makes the rules useless.

Why robots.txt for AI crawlers is more than one rule

User-agent: * used to be enough. With AI it is too coarse, because the same company runs different bots for different jobs. OpenAI says so itself: each setting is independent of the others. You can allow OAI-SearchBot so that your site can appear in ChatGPT's search results, and disallow GPTBot so that your content is not meant for training OpenAI's foundation models (OpenAI).

That means blocking only GPTBot in the hope of disappearing from ChatGPT does something different from what you expect. And blocking everything out of caution also shuts out the bots that could cite you in answers.

The three kinds of AI crawlers

Kind	Bots	What happens if you block them
Training	GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google)	You signal that your content should not be used for training. Google Search is not affected
Search	OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), PerplexityBot (Perplexity)	Your site is not shown, or shown less, in that service's search answers
On a user's request	ChatGPT-User (OpenAI), Claude-User (Anthropic)	For OpenAI, robots.txt rules may not apply because a person asked. Anthropic says you can control Claude-User, and that blocking it may reduce visibility in user-directed search

What the operators write themselves, in short:

OpenAI: sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, but can still appear as navigational links. After a robots.txt change it can take about 24 hours until the systems adjust.
Anthropic: if you restrict ClaudeBot, that signals that future material should be excluded from training datasets. If you disable Claude-SearchBot or Claude-User, your visibility in search results may drop. Anthropic honors robots.txt, does not bypass CAPTCHAs, and supports the non-standard Crawl-delay line (Anthropic).
Perplexity: PerplexityBot is designed to surface and link websites in Perplexity's search results. It is not used to crawl content for foundation models, and Perplexity recommends allowing it (Perplexity).
Google: Google-Extended is not a separate bot but a token in robots.txt. It controls whether content Google crawls may be used to train future Gemini models and for grounding. It does not affect inclusion in Google Search and is not a ranking signal (Google).

Operators change their lists. Check these pages before you copy a rule.

Three examples you can adapt

1. Allow everything, keep only the internal API closed. This is the approach of seogrinder.com itself: one group with several User-agent lines above one set of rules.

User-agent: *
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
Disallow: /api/
Allow: /

Sitemap: https://www.example.com/sitemap.xml

2. Allow search, refuse training.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

This is the owner's choice, not a recommendation. If you want your content used as widely as possible, take example 1. Note that Google-Extended also covers grounding in Gemini Apps and on Vertex AI, not only training. It does not affect Google Search or AI Overviews.

3. Refuse all AI bots. Put Disallow: / under every bot listed above. You will then appear less often, or not at all, in the search answers of ChatGPT, Claude and Perplexity (navigational links and fetches on a user's request remain possible). Google's AI Overviews are controlled through Googlebot, not through these tokens. Only sensible if that is what you want.

The trap: a bot's own group replaces the `*` group

According to RFC 9309, a crawler follows the group that matches its name. If it has a group of its own, the * group does not apply to it (RFC 9309).

An example: under User-agent: * you have Disallow: /internal/. Later you add a separate group for GPTBot that contains only Allow: /. Now /internal/ is no longer off limits for GPTBot. Repeat the rules in every group, or put several User-agent lines above the same set of rules, as in the examples above.

Two more rules from the same specification help when rules conflict: the most specific match wins (the longest path), and if an Allow and a Disallow are equally long, Allow is used.

An optional line that states how your content may be used: Content-Signal

Besides allowing and blocking, there is an optional line that Cloudflare introduced in September 2025. It states what your content may be used for:

Content-Signal: search=yes, ai-input=yes, ai-train=no

search covers search indexes and snippets, ai-input covers using content directly in AI answers, and ai-train covers training. It is a declaration, not a lock: compliance is voluntary. Cloudflare says that content signals express preferences, that they are not technical countermeasures, and that some companies might simply ignore them (Cloudflare).

Standards work is also under way. Google's crawler documentation notes that Google intends to support the AIPREF shared, machine-readable training preference signal in early 2027, written as user-agent: __usage_ai-training (Google). Until then, Google-Extended stays the switch for Gemini training and grounding.

robots.txt is not security

The specification says it plainly: the protocol is not a substitute for real security measures, and paths listed in robots.txt are visible to everyone. A bot that ignores the rules is not stopped by your file. For content that must not be public, use a login or firewall rules. Anthropic also warns that blocking by IP address may not work as an opt-out, because it stops its bot from reading your robots.txt in the first place.

How to check your robots.txt for AI crawlers

Open https://yoursite.com/robots.txt in a browser. Every subdomain has its own file.
Open SEOgrinder, enter your address and start the analysis. No account is needed.
Go to the AI readiness tab and find "Steer AI bots" under "Checks for AI agents". If the point is met, the line says how many AI crawlers are governed, how many are allowed and how many are blocked. If it is open, no explicit rule for training and search was found.
A met point can be expanded: it shows the text that was found in your file.
Press Copy agent prompt and paste it into your coding assistant. If a rule is missing, the prompt describes what to add; if one exists, it asks the assistant to review it and change only what is wrong. Allowing or blocking stays the owner's decision.
After the change, press Check again. Reports are cached for up to 24 hours; this button forces a fresh run (a few times per hour).

Plan for about a day before OpenAI's and Perplexity's systems apply new rules.

Spotting real visits: logs and IP ranges

Whether a bot really visits your site shows up in your server logs, in the user agent. Because anyone can call themselves GPTBot, the operators publish IP ranges for comparison: OpenAI lists them for each bot as a JSON file, for example openai.com/gptbot.json, and Anthropic points to a list of its own. If a visit does not come from those ranges, it is very likely not from that operator, even if the name matches. That is useful if you want to analyse visits or catch fake bots in a firewall. Keep Anthropic's warning in mind: do not block the real bots by IP address, or they can no longer read your robots.txt.

Common mistakes

Leaving Disallow: / in place for everyone. A staging site goes live with the staging block still in its robots.txt. Check this at every launch.
Blocking only GPTBot to leave ChatGPT. That controls training, not search.
A blank line inside a group. Some parsers, such as Python's robotparser, then drop the names above it. Keep the User-agent lines directly together.
A relative Sitemap line. Write the full address with https://.
The file in the wrong place. It has to be at the root: /robots.txt, not in a subfolder.
Expecting too much from Content-Signal and llms.txt. Both are optional, and an effect on ranking or citations is not proven.

FAQ

Should I allow AI crawlers?

It depends on your goal. If you want to be citable in AI search answers, allow the search bots. Allowing or refusing training is a business decision, and you can set the two independently.

Will I disappear from Google if I block Google-Extended?

No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal.

How fast does a change work?

OpenAI and Perplexity both mention up to about 24 hours after your robots.txt changes.

Is llms.txt the same as robots.txt?

No. robots.txt controls access. llms.txt is a proposed file with a summary of your site for language models. According to second-hand reports, Google does not support it (Search Engine Roundtable). An effect is not proven.

Can I set different rules for different folders?

Yes. Inside a group you can set Disallow and Allow per path. The most specific match wins.

Your next step

Open your robots.txt and check that you have not blocked the search bots by accident. Then run the free scan and see whether "Steer AI bots" is met under AI readiness, which means that training and search each have an explicit rule. Whether the rule allows or blocks is shown in the finding.