How to get cited by Claude: 3 Anthropic bots and which to allow

By SEOgrinder · Published November 4, 2026 · 9 min read

Short answer: Nobody can promise that you will be cited by Claude. Anthropic's help page names three robots and says what happens when you disable each one; it does not describe how Claude picks the pages it names. What you control is robots.txt. ClaudeBot collects content that may contribute to training, Claude-User fetches a page when someone asks Claude a question, and Claude-SearchBot works on the quality of search results. For the last two, Anthropic says that disabling them may reduce your visibility in search. Training is a separate decision.

What you control when you want to be cited by Claude

Start with the limit. Whether Claude names a page in an answer is Claude's decision. Anthropic's help article "Does Anthropic crawl data from the web, and how can site owners block the crawler?" is about crawling and opting out. It says which robots exist, what each is for and how to limit them. It does not say how Claude chooses sources, whether allowing a bot leads to a mention, or how long a change takes. We know of no source that proves an effect of a robots.txt setting on citations, so this article promises none.

What the page does give you is a split of jobs. Anthropic says it uses robots "to gather data from the public web for model development, to search the web, and to retrieve web content at users’ direction". The page is dated 7 April 2026, and we fetched it on 4 November 2026. Everything we say about Anthropic comes from it, and we mark our own readings.

Readable is not the same as cited. A page that Anthropic's bots can read can still be left out of an answer, and a page they cannot read gives them nothing to use. Readability is the part you control. SEOgrinder checks that part and does not measure citations; our article on the AI visibility checker explains the difference.

The three Anthropic bots in one table

Bot	What Anthropic says it is for	What Anthropic says happens when you disable it
ClaudeBot	Collects web content that could potentially contribute to the training of its generative AI models	Signals that the site's future materials should be excluded from the training datasets
Claude-User	Supports Claude users: when someone asks Claude a question, it may access websites with this agent	Stops Anthropic from retrieving your content in response to a user query, which may reduce your visibility for user-directed web search
Claude-SearchBot	Navigates the web to improve search result quality for users, by analysing content to make search responses more relevant and accurate	Stops Anthropic from indexing your content for search optimization, which may reduce your visibility and accuracy in user search results

The same page adds that its bots honor the industry-standard directives in robots.txt and respect anti-circumvention technologies: it will not try to bypass CAPTCHAs. It gives no recommendation to allow any bot. It describes what happens when you disable one, and the choice stays with you.

Bot 1: ClaudeBot, the training bot

According to the table, restricting ClaudeBot signals that your future material should be excluded from Anthropic's training datasets. The page speaks of future material, and we do not read more into it.

What the row leaves out matters here. For ClaudeBot, Anthropic names the training exclusion and no effect on search; the visibility effects appear in the rows of the other two bots. That is our reading of a table, not a statement by Anthropic, but it suggests that ClaudeBot is not the setting to look at if you want to be found in Claude's search. Whether to allow training is a business decision that SEOgrinder does not make for you.

Bot 2: Claude-User, the bot that fetches when someone asks

Claude-User is the bot behind a single request: a person asks Claude a question, and Claude may open a website to answer it. Anthropic writes "may", so a question does not always lead to a visit.

Anthropic says Claude-User lets site owners control which sites can be accessed through these user-initiated requests. Together with its statement that all its bots honor robots.txt, that means a rule in your file applies here too. Other operators differ: OpenAI writes that for ChatGPT-User "robots.txt rules may not apply" (OpenAI). Read each operator's page and do not carry a rule over from one to the other.

If you disable Claude-User, Anthropic says its system can no longer retrieve your content in response to a user query, which may reduce your visibility for user-directed web search.

Bot 3: Claude-SearchBot, the search bot

Claude-SearchBot is the robot whose description is about search results themselves. Anthropic says it navigates the web to improve search result quality and analyses content to make search responses more relevant and accurate. Disabling it prevents indexing of your content for search optimization.

Our reading: if you want to appear in Claude's search results, Claude-SearchBot is the first line to check in your robots.txt, because its job is indexing for search. Allowing it is the precondition for what the page describes. It is not a promise that anything will be shown.

Which bot to allow? Three questions

Anthropic does not say which bot to allow, so here is how we would decide. This is our opinion, not Anthropic's.

Do you want your pages in Claude's search results? Then do not disable Claude-SearchBot.
Do you want Claude to open your pages when a person asks about them? Then do not disable Claude-User.
Do you want your content in training datasets? That is the ClaudeBot decision, and it can differ from the first two. A site can allow all three or refuse only training.

None of the answers is a technical error. A mistake happens when you decide by accident, for example because a Disallow: / from a staging site is still in your file.

How to allow or block each bot in robots.txt

A file that allows search and user fetches and refuses training:

# Training: your own decision (this example refuses it)
User-agent: ClaudeBot
Disallow: /

# Search and fetches on a user's request
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

To block a bot from your whole site, Anthropic's page shows a User-agent line with the bot's name followed by Disallow: /, placed in the robots.txt of your top-level directory. It asks you to do this for every subdomain you want to opt out from, so www.example.com and blog.example.com each need their own file. To slow a bot down instead, the page says Anthropic supports the non-standard Crawl-delay line and shows Crawl-delay: 1 for ClaudeBot.

The page also warns against blocking the IP addresses of its bots. That may not work correctly or guarantee an opt-out, because it stops Anthropic from reading your robots.txt. Put the rule in robots.txt.

One trap comes from the standard, not from Anthropic. According to RFC 9309, a crawler follows the group that matches its name, and if it has one of its own, the * group does not apply to it. If you add a group for Claude-SearchBot, repeat every rule that must still hold. Our article on robots.txt for AI crawlers shows the fix and covers OpenAI and Perplexity too.

Our own file is an example, not a recommendation. On 4 November 2026 we fetched https://seogrinder.com/robots.txt with curl: one shared group with * and nine AI user agents, among them ClaudeBot, Claude-SearchBot and Claude-User, allowing everything except /api/. Yours can differ.

Telling real Anthropic visits from fakes

The page links a list, claude.com/crawling/bots.json, and says that a crawler with a source address on it is coming from Anthropic. We fetched it on 4 November 2026: one JSON file with address prefixes and a creationTime field that read 20 October 2026. It is not split by bot, so it shows that a visit is from Anthropic, not which of the three made it. That you can only read from the name in the user agent, and anyone can write a name into a header, so count a visit only if its address is on the list. Fetch the file again from time to time.

Search your log for the names, not for a version. The page does not print full user-agent strings. An empty log is no proof of a block, because the page does not say how often the bots visit.
A bot that gets a challenge page reads nothing. Anthropic says its bots will not bypass CAPTCHAs. Whether they run JavaScript is not on its page; our article on AI crawlers and JavaScript covers what is known.

How SEOgrinder helps

SEOgrinder is free and needs no account, and we build it. Enter an address, start the analysis and open the report. Two places matter here:

AI readiness tab, "Checks for AI agents": the point "Steer AI bots" shows how many AI crawlers have an explicit rule, how many are allowed and how many are blocked, and names the blocked ones. It knows ClaudeBot, Claude-User and Claude-SearchBot by name. ClaudeBot counts as a training bot and Claude-SearchBot as a search bot, and the point is met only when training and search each have an explicit rule. A rule for Claude-User alone settles neither.
Tasks tab, "Server may turn away allowed AI crawlers": SEOgrinder tests how the server answers four fixed bot names, only those that robots.txt allows, and compares it with a browser. Of Anthropic's bots, only ClaudeBot is among them, and the test uses only the name.

Press Copy agent prompt at a point to get an instruction for your coding assistant, and Check again after a change. SEOgrinder does not read your server log and does not measure whether Claude names your site.

Common mistakes

Blocking ClaudeBot to leave Claude's search. For ClaudeBot, Anthropic names the training exclusion. The search effects are named for the other two bots.
Blocking all three out of caution. For Claude-User and Claude-SearchBot, Anthropic says visibility may drop.
Blocking by IP address instead of in robots.txt. Anthropic says this may not work.
Fixing only the main domain. Anthropic asks for the rule on every subdomain you want to opt out.
Adding a group for one bot and losing the shared rules. Its own group replaces the * group.
Reading an allowed bot as a citation. Allowing a bot removes a blocker and promises no mention.

FAQ

Can I make Claude cite my page?

No. Nobody can force a citation, and Anthropic's page describes no setting that causes one. You can remove blockers so that its bots can read your pages. Whether Claude names them is its decision.

Does blocking ClaudeBot remove me from Claude's search?

Anthropic's page does not say so. For ClaudeBot it names the exclusion of future material from training datasets. The search effects it names belong to Claude-User and Claude-SearchBot.

Does SEOgrinder show whether Claude cites my site?

No. It checks the technical basis: whether AI systems can reach and read the page.

Your next step

Open your robots.txt and look for ClaudeBot, Claude-User and Claude-SearchBot. Decide each one on purpose: search and user fetches on one side, training on the other. If you run subdomains, check each file. Then run the free scan in SEOgrinder, open the AI readiness tab and the Tasks tab, and fix what is open.