Free tool · no signup

AI robots.txt generator

Decide which AI crawlers may read your site, one by one. Pick allow or block for each, and copy robots.txt rules that say exactly that. A common choice is to stay in AI search answers while opting out of model training, so that is the starting preset.

Start from a preset

The first preset leaves Google-Extended allowed. Google documents that the token covers grounding in Gemini as well as training, so blocking it is more than a training opt-out.

  • OAI-SearchBot Live OpenAI

    The crawler that decides whether your pages can appear as sources in ChatGPT search answers.

  • ChatGPT-User Live OpenAI

    The user agent ChatGPT uses when a person asks it to open or read a page.

  • GPTBot Training OpenAI

    OpenAI's training crawler: it collects pages that may be used to train future models.

  • Claude-SearchBot Live Anthropic

    Anthropic's search crawler, which indexes pages to improve Claude's search results.

  • Claude-User Live Anthropic

    The user agent Claude uses to fetch a page when a user's question needs it.

  • ClaudeBot Training Anthropic

    Anthropic's training crawler: it collects web content that could contribute to training Claude models.

  • PerplexityBot Live Perplexity

    Perplexity's crawler for surfacing and linking websites in its search results.

  • Perplexity-User Live Perplexity

    The user agent Perplexity uses when a user's question leads it to visit a page.

  • Google-Extended Control token Google

    Not a crawler: a robots.txt token that controls whether Google may use your pages for Gemini.

  • CCBot Training Common Crawl

    The crawler behind Common Crawl, an open web archive that many AI models have been trained on.

Your robots.txt rules

Add these to the robots.txt you already have. Do not replace your existing User-agent: * rules with them.

A crawler follows only the most specific group that names it. Once a crawler is named here it stops reading your User-agent: * group, so list any private paths above and they are repeated for the allowed crawlers.

Rules written. Now check what your live site returns.

A robots.txt rule is only half of it: a firewall or CDN can still refuse a crawler the file allows. A free SEO Sets account runs the crawler check on your own site and keeps the history, so you can see when access changes.

Track AI crawler access: start free

No account needed for a one-off test: run the AI crawler check on your live site

How to choose: live crawlers and training crawlers

AI companies run more than one crawler, and each does a different job. Treating them as one thing is how sites end up invisible in AI answers without meaning to be. The 10 tokens above are the ones our AI crawler check reads, and they fall into two groups.

  • Live crawlers fetch pages to answer people now. OAI-SearchBot, Claude-SearchBot and PerplexityBot build the search results that ChatGPT, Claude and Perplexity cite. ChatGPT-User, Claude-User and Perplexity-User fetch a page when a person's question needs it.
  • Training crawlers collect pages that may be used to train future models: GPTBot, ClaudeBot and CCBot. Google-Extended is a robots.txt control token, not a crawler. It decides whether Google may use your pages for Gemini training and grounding.

The two groups are independent. Blocking GPTBot opts you out of OpenAI's training and, per OpenAI, leaves ChatGPT search alone, because search is governed by OAI-SearchBot. Block OAI-SearchBot and OpenAI says your site will not be shown in ChatGPT search answers. The same split applies to Anthropic's ClaudeBot and Claude-SearchBot.

What robots.txt can and cannot do

robots.txt is a request that well-behaved crawlers follow. It is not a lock. Two of the user-triggered agents are documented by their own operators as exceptions: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page. A rule for those two records your preference. Only a server or CDN rule stops the visit itself.

It also works the other way. A robots.txt that allows every crawler does not help if your firewall answers them with a 403. That is why the next step after pasting these rules is to test the live site, not to assume the file is the whole story.

Where the rules go

robots.txt lives at the root of each host, so example.com and www.example.com each need their own copy. Paste the generated groups below your existing rules. Changes are not instant: OpenAI and Perplexity both say a robots.txt change can take up to about 24 hours to apply. Blocking a training crawler stops future collection. It does not remove pages that were already collected.

Crawler facts on this page come from each operator's own documentation, read on November 17, 2026, and each crawler links to its page in our AI crawler directory with the source.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT answers?

No. GPTBot is OpenAI's training crawler. OpenAI documents that ChatGPT search is governed by a separate crawler, OAI-SearchBot, and treats the two settings independently. Blocking OAI-SearchBot is what takes a site out of ChatGPT search answers.

Which AI crawlers should I allow?

If you want AI assistants to cite you, allow the live crawlers: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User. Whether to allow the training crawlers (GPTBot, ClaudeBot, CCBot) and Google-Extended is a policy choice about model training, and it does not decide whether you are cited today.

Do AI crawlers obey robots.txt?

The search and training crawlers in this list are documented by their operators as following robots.txt. Two user-triggered agents are documented exceptions: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. For those, only a server or CDN rule stops the visit.

Does this tool read or change my site?

No. The rules are built in your browser from the choices you make, and nothing you type is sent to SEO Sets. To see what your live robots.txt and server actually return to each crawler, run the free AI crawler check.

More free tools

SEO Sets Support

Press Enter to send · Shift+Enter for new line