AI crawler · Anthropic
Claude-User
The user agent Claude uses to fetch a page when a user's question needs it.
- Operator
- Anthropic
- Feeds
- Claude
- Type
- Live (answers users now)
- robots.txt token
- Claude-User
What Claude-User does
Anthropic says Claude-User supports Claude users when they ask questions that require accessing a website.
Does it follow robots.txt? Yes. Anthropic says its bots, including this one, honour robots.txt directives.
robots.txt: allow or block Claude-User
Put the group in the robots.txt at the root of every host it should cover
(example.com and www.example.com each need their own).
Allow
User-agent: Claude-User Allow: /
Block
User-agent: Claude-User Disallow: /
Setting rules for several AI crawlers at once? The free AI robots.txt generator writes the groups for all of them.
User-agent: Claude-User group, it ignores your User-agent: * rules, so repeat
any Disallow lines (admin areas, carts) you still want applied to it.
What blocking does and does not do
- Does: Per Anthropic, it stops Claude retrieving your content for user queries, which may reduce your visibility in user-directed web search.
- Does not: It does not affect training (ClaudeBot) or Anthropic's search index (Claude-SearchBot).
Check whether your site blocks Claude-User
Enter your page and our free crawler check opens with it filled in. It reads your robots.txt for Claude-User and the other nine AI crawler tokens, then requests the page as Claude-User to catch a server or CDN refusal. No signup.
Frequently asked questions
How do I block Claude-User in robots.txt?
Add a group that names it: "User-agent: Claude-User" on one line and "Disallow: /" on the next, in the robots.txt at the root of each host you want covered.
Does blocking Claude-User remove my site from AI answers?
Per Anthropic, it stops Claude retrieving your content for user queries, which may reduce your visibility in user-directed web search. It does not affect training (ClaudeBot) or Anthropic's search index (Claude-SearchBot).
Is Claude-User a live crawler or a training crawler?
Live: it fetches pages to answer users now, which is the path to being cited. Does it follow robots.txt? Yes. Anthropic says its bots, including this one, honour robots.txt directives.
How can I check whether my site blocks Claude-User?
Run the free SEO Sets AI crawler check. It reads your robots.txt rules for this and nine other AI crawler tokens, and for real user agents it also requests your page with that name to see whether your server or CDN refuses it.
Official documentation
Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?. Facts on this page were read from it on November 17, 2026. Vendors change their crawlers; the vendor's page wins if it disagrees with ours.
Other Anthropic crawlers: Claude-SearchBot, ClaudeBot · All AI crawlers
Related
- AI crawler checkFree, no signup: is any AI crawler blocked on your site?
- AI robots.txt generatorFree: allow or block each AI crawler and copy the rules.
- AI crawler directoryEvery AI crawler our check reads, with robots.txt rules.
- Claim your business in ChatGPTA first-hand, step-by-step guide.
- Compare SEO SetsDated, sourced comparisons with Otterly and Peec.
See what to fix on your own site
The free Signal plan connects your Google Search Console and shows your top 3 pages with the most search upside, with no time limit and no credit card. AI citation tracking starts on Surge at $49 a month.