Free tool · no signup
AI robots.txt generator
Decide which AI crawlers may read your site, one by one. Pick allow or block for each, and copy robots.txt rules that say exactly that. A common choice is to stay in AI search answers while opting out of model training, so that is the starting preset.
Start from a preset
The first preset leaves Google-Extended allowed. Google documents that the token covers grounding in Gemini as well as training, so blocking it is more than a training opt-out.
-
OAI-SearchBot Live OpenAI
The crawler that decides whether your pages can appear as sources in ChatGPT search answers.
-
ChatGPT-User Live OpenAI
The user agent ChatGPT uses when a person asks it to open or read a page.
-
GPTBot Training OpenAI
OpenAI's training crawler: it collects pages that may be used to train future models.
-
Claude-SearchBot Live Anthropic
Anthropic's search crawler, which indexes pages to improve Claude's search results.
-
Claude-User Live Anthropic
The user agent Claude uses to fetch a page when a user's question needs it.
-
ClaudeBot Training Anthropic
Anthropic's training crawler: it collects web content that could contribute to training Claude models.
-
PerplexityBot Live Perplexity
Perplexity's crawler for surfacing and linking websites in its search results.
-
Perplexity-User Live Perplexity
The user agent Perplexity uses when a user's question leads it to visit a page.
-
Google-Extended Control token Google
Not a crawler: a robots.txt token that controls whether Google may use your pages for Gemini.
-
CCBot Training Common Crawl
The crawler behind Common Crawl, an open web archive that many AI models have been trained on.
Your robots.txt rules
Add these to the robots.txt you already have. Do not replace your existing
User-agent: * rules with them.
What each block does
User-agent: * group, so list any private paths above and they
are repeated for the allowed crawlers.
Rules written. Now check what your live site returns.
A robots.txt rule is only half of it: a firewall or CDN can still refuse a crawler the file allows. A free SEO Sets account runs the crawler check on your own site and keeps the history, so you can see when access changes.
Track AI crawler access: start freeNo account needed for a one-off test: run the AI crawler check on your live site
How to choose: live crawlers and training crawlers
AI companies run more than one crawler, and each does a different job. Treating them as one thing is how sites end up invisible in AI answers without meaning to be. The 10 tokens above are the ones our AI crawler check reads, and they fall into two groups.
- Live crawlers fetch pages to answer people now. OAI-SearchBot, Claude-SearchBot and PerplexityBot build the search results that ChatGPT, Claude and Perplexity cite. ChatGPT-User, Claude-User and Perplexity-User fetch a page when a person's question needs it.
- Training crawlers collect pages that may be used to train future models: GPTBot, ClaudeBot and CCBot. Google-Extended is a robots.txt control token, not a crawler. It decides whether Google may use your pages for Gemini training and grounding.
The two groups are independent. Blocking GPTBot opts you out of OpenAI's training and, per OpenAI, leaves ChatGPT search alone, because search is governed by OAI-SearchBot. Block OAI-SearchBot and OpenAI says your site will not be shown in ChatGPT search answers. The same split applies to Anthropic's ClaudeBot and Claude-SearchBot.
What robots.txt can and cannot do
robots.txt is a request that well-behaved crawlers follow. It is not a lock. Two of the user-triggered agents are documented by their own operators as exceptions: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page. A rule for those two records your preference. Only a server or CDN rule stops the visit itself.
It also works the other way. A robots.txt that allows every crawler does not help if your firewall answers them with a 403. That is why the next step after pasting these rules is to test the live site, not to assume the file is the whole story.
Where the rules go
robots.txt lives at the root of each host, so example.com and
www.example.com each need their own copy. Paste the generated groups below your
existing rules. Changes are not instant: OpenAI and Perplexity both say a robots.txt change can take
up to about 24 hours to apply. Blocking a training crawler stops future collection. It does not remove
pages that were already collected.
Crawler facts on this page come from each operator's own documentation, read on November 17, 2026, and each crawler links to its page in our AI crawler directory with the source.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT answers?
No. GPTBot is OpenAI's training crawler. OpenAI documents that ChatGPT search is governed by a separate crawler, OAI-SearchBot, and treats the two settings independently. Blocking OAI-SearchBot is what takes a site out of ChatGPT search answers.
Which AI crawlers should I allow?
If you want AI assistants to cite you, allow the live crawlers: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User. Whether to allow the training crawlers (GPTBot, ClaudeBot, CCBot) and Google-Extended is a policy choice about model training, and it does not decide whether you are cited today.
Do AI crawlers obey robots.txt?
The search and training crawlers in this list are documented by their operators as following robots.txt. Two user-triggered agents are documented exceptions: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. For those, only a server or CDN rule stops the visit.
Does this tool read or change my site?
No. The rules are built in your browser from the choices you make, and nothing you type is sent to SEO Sets. To see what your live robots.txt and server actually return to each crawler, run the free AI crawler check.
More free tools
- AI crawler checkIs any AI crawler blocked by your robots.txt or your server?
- AI readiness scoreA 0 to 100 score for how easy a page is for an AI engine to quote.
- Does ChatGPT cite my site?One live buyer question, and the sources the answer cited.
- llms.txt generatorTurn your site name, a one-sentence summary and your key pages into a correctly formatted llms.txt to copy or download.
- Title and meta description checkerType a title and description, see a live search result preview, character counts and plain warnings before you publish.
- LocalBusiness and FAQ schema generatorFill in a short form for a local business or a set of questions and answers, and copy valid JSON-LD in a script tag.