Free tool · no signup

AI crawler check

Is your site letting ChatGPT, Claude, Perplexity and Gemini in? Enter a page URL and we check your robots.txt rules for each AI crawler, then request the page with each crawler's name in the user-agent to see whether your server or CDN turns it away.

Takes up to about 30 seconds: we read robots.txt and llms.txt, then make about ten requests to the page you enter.

What this check does

AI assistants read the web through their own crawlers, and each crawler obeys the robots.txt group that names it. A single Disallow: / under User-agent: GPTBot, or a firewall rule that refuses anything calling itself a bot, can keep a site out of AI answers without anyone noticing.

  • robots.txt, whole site — whether the crawler's group (or the * group) blocks the site root.
  • robots.txt, this URL — the same rules applied to the exact path you entered, using longest-match Allow/Disallow with * and $ wildcards.
  • Server response — the HTTP status your server returned when the request carried that crawler's name in its user-agent, compared with a request carrying our own.

We follow up to 3 redirects, refuse private and internal addresses, and read at most 2 MB per page. No AI model is called. The full method for every number SEO Sets shows is on How we measure.

Frequently asked questions

Which AI crawlers does this check?

OpenAI (OAI-SearchBot, ChatGPT-User, GPTBot), Anthropic (Claude-SearchBot, Claude-User, ClaudeBot), Perplexity (PerplexityBot, Perplexity-User), Google-Extended (the robots.txt token that controls Gemini training and grounding use) and Common Crawl (CCBot).

What is the difference between a live and a training crawler?

Live crawlers fetch pages to answer a user right now, which is the path to being cited in an answer. Training crawlers collect pages that may shape future models. Blocking a training crawler does not stop a live one, and the reverse is also true.

How does the check work?

We read your robots.txt and apply each crawler's rules to the site root and to the exact URL you entered. We then request that URL once with our own user-agent and once per crawler with a user-agent that contains the crawler's name, and show the HTTP status each request got. A crawler that robots.txt allows but your server answers with 403 is still blocked in practice.

Are these requests really from GPTBot or ClaudeBot?

No. They come from SEO Sets, and the user-agent says so. Some CDNs and firewalls verify crawlers by IP address, so they can treat the real crawler differently from our test. A refusal here is strong evidence of a user-agent rule; a success here does not prove the real crawler gets through.

Does blocking Google-Extended remove my site from Google Search?

No. Google-Extended only controls whether Google may use your content for Gemini training and grounding. Google Search crawling is governed by Googlebot, which this tool does not change or test.

Does an llms.txt file help?

We report whether /llms.txt exists, as information only. Whether the AI engines actually use it is unproven, so it does not change any verdict on this page.

Want this tracked for your own site?

A free SEO Sets account connects your Google Search Console, keeps a history of these checks for your own site so you can see whether a fix moved the score, and shows your top 3 pages with the most search upside. No credit card.

More free tools

What each crawler does, and the robots.txt rule to allow or block it: AI crawler directory.

SEO Sets Support

Press Enter to send · Shift+Enter for new line