Free tool · no signup
AI crawler check
Is your site letting ChatGPT, Claude, Perplexity and Gemini in? Enter a page URL and we check your robots.txt rules for each AI crawler, then request the page with each crawler's name in the user-agent to see whether your server or CDN turns it away.
What this check does
AI assistants read the web through their own crawlers, and each crawler obeys the
robots.txt group that names it. A single Disallow: / under
User-agent: GPTBot, or a firewall rule that refuses anything calling itself a bot, can keep a
site out of AI answers without anyone noticing.
- robots.txt, whole site — whether the crawler's group (or the
*group) blocks the site root. - robots.txt, this URL — the same rules applied to the exact path you entered, using longest-match Allow/Disallow with
*and$wildcards. - Server response — the HTTP status your server returned when the request carried that crawler's name in its user-agent, compared with a request carrying our own.
We follow up to 3 redirects, refuse private and internal addresses, and read at most 2 MB per page. No AI model is called. The full method for every number SEO Sets shows is on How we measure.
Frequently asked questions
Which AI crawlers does this check?
OpenAI (OAI-SearchBot, ChatGPT-User, GPTBot), Anthropic (Claude-SearchBot, Claude-User, ClaudeBot), Perplexity (PerplexityBot, Perplexity-User), Google-Extended (the robots.txt token that controls Gemini training and grounding use) and Common Crawl (CCBot).
What is the difference between a live and a training crawler?
Live crawlers fetch pages to answer a user right now, which is the path to being cited in an answer. Training crawlers collect pages that may shape future models. Blocking a training crawler does not stop a live one, and the reverse is also true.
How does the check work?
We read your robots.txt and apply each crawler's rules to the site root and to the exact URL you entered. We then request that URL once with our own user-agent and once per crawler with a user-agent that contains the crawler's name, and show the HTTP status each request got. A crawler that robots.txt allows but your server answers with 403 is still blocked in practice.
Are these requests really from GPTBot or ClaudeBot?
No. They come from SEO Sets, and the user-agent says so. Some CDNs and firewalls verify crawlers by IP address, so they can treat the real crawler differently from our test. A refusal here is strong evidence of a user-agent rule; a success here does not prove the real crawler gets through.
Does blocking Google-Extended remove my site from Google Search?
No. Google-Extended only controls whether Google may use your content for Gemini training and grounding. Google Search crawling is governed by Googlebot, which this tool does not change or test.
Does an llms.txt file help?
We report whether /llms.txt exists, as information only. Whether the AI engines actually use it is unproven, so it does not change any verdict on this page.
Want this tracked for your own site?
A free SEO Sets account connects your Google Search Console, keeps a history of these checks for your own site so you can see whether a fix moved the score, and shows your top 3 pages with the most search upside. No credit card.
More free tools
- AI readiness scoreA 0 to 100 score for how easy a page is for an AI engine to quote.
- Does ChatGPT cite my site?One live buyer question, and the sources the answer cited.
- AI robots.txt generatorPick allow or block for each AI crawler and copy the robots.txt rules, with a plain summary of what your choice changes.
- llms.txt generatorTurn your site name, a one-sentence summary and your key pages into a correctly formatted llms.txt to copy or download.
- Title and meta description checkerType a title and description, see a live search result preview, character counts and plain warnings before you publish.
- LocalBusiness and FAQ schema generatorFill in a short form for a local business or a set of questions and answers, and copy valid JSON-LD in a script tag.
What each crawler does, and the robots.txt rule to allow or block it: AI crawler directory.