AI crawler · Anthropic

Claude-SearchBot

Anthropic's search crawler, which indexes pages to improve Claude's search results.

Operator
Anthropic
Feeds
Claude search
Type
Live (answers users now)
robots.txt token
Claude-SearchBot

What Claude-SearchBot does

Anthropic says Claude-SearchBot navigates the web to improve search result quality for Claude users.

Does it follow robots.txt? Yes. Anthropic says its bots honour robots.txt directives.

How many popular sites block Claude-SearchBot

11.3% of the Tranco top 500 sites whose robots.txt answered block Claude-SearchBot for the whole site (36 of 319), as of . The number comes from our weekly AI crawler access study, which counts a site as blocking only when its robots.txt disallows the site root for this token.

robots.txt: allow or block Claude-SearchBot

Put the group in the robots.txt at the root of every host it should cover (example.com and www.example.com each need their own).

Allow

User-agent: Claude-SearchBot
Allow: /

Block

User-agent: Claude-SearchBot
Disallow: /

Setting rules for several AI crawlers at once? The free AI robots.txt generator writes the groups for all of them.

A crawler follows only the most specific group that names it. Once you add a User-agent: Claude-SearchBot group, it ignores your User-agent: * rules, so repeat any Disallow lines (admin areas, carts) you still want applied to it.

What blocking does and does not do

  • Does: Per Anthropic, blocking it stops your content being indexed for search, which may reduce your visibility and accuracy in Claude's search results.
  • Does not: It does not opt you out of training (that is ClaudeBot) and it does not cover user-requested fetches (that is Claude-User).
  • Anthropic publishes the token names but not full user-agent strings, and says blocking its IP addresses may not reliably opt you out. Use robots.txt.

Check whether your site blocks Claude-SearchBot

Enter your page and our free crawler check opens with it filled in. It reads your robots.txt for Claude-SearchBot and the other nine AI crawler tokens, then requests the page as Claude-SearchBot to catch a server or CDN refusal. No signup.

Frequently asked questions

How do I block Claude-SearchBot in robots.txt?

Add a group that names it: "User-agent: Claude-SearchBot" on one line and "Disallow: /" on the next, in the robots.txt at the root of each host you want covered.

Does blocking Claude-SearchBot remove my site from AI answers?

Per Anthropic, blocking it stops your content being indexed for search, which may reduce your visibility and accuracy in Claude's search results. It does not opt you out of training (that is ClaudeBot) and it does not cover user-requested fetches (that is Claude-User).

Is Claude-SearchBot a live crawler or a training crawler?

Live: it fetches pages to answer users now, which is the path to being cited. Does it follow robots.txt? Yes. Anthropic says its bots honour robots.txt directives.

How can I check whether my site blocks Claude-SearchBot?

Run the free SEO Sets AI crawler check. It reads your robots.txt rules for this and nine other AI crawler tokens, and for real user agents it also requests your page with that name to see whether your server or CDN refuses it.

Official documentation

Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?. Facts on this page were read from it on November 17, 2026. Vendors change their crawlers; the vendor's page wins if it disagrees with ours.

Other Anthropic crawlers: Claude-User, ClaudeBot · All AI crawlers

See what to fix on your own site

The free Signal plan connects your Google Search Console and shows your top 3 pages with the most search upside, with no time limit and no credit card. AI citation tracking starts on Surge at $49 a month.

SEO Sets Support

Press Enter to send · Shift+Enter for new line