Research · updated weekly

Which popular sites block AI crawlers?

Every week we read the robots.txt of the same 500 popular domains (Tranco top 500, a public research ranking) and record which AI crawlers each one blocks. The numbers below update on their own.

Last updated · 319 of 500 domains answered

Share of sites whose robots.txt blocks each crawler

17.2%

GPTBot

55 of 319 sites

16.6%

ClaudeBot

53 of 319 sites

13.5%

PerplexityBot

43 of 319 sites

15%

Google-Extended

48 of 319 sites

18.5%

CCBot

59 of 319 sites

10.3%

OAI-SearchBot

33 of 319 sites

21% (67 of 319) block at least one of the 8 AI crawlers we track for the whole site. 231 domains served a robots.txt file; the other answering domains had none (which allows everything).

All tracked crawlers, latest run
CrawlerSites blockingShare
GPTBot 55 17.2%
OAI-SearchBot 33 10.3%
ChatGPT-User 35 11%
ClaudeBot 53 16.6%
Claude-SearchBot 36 11.3%
PerplexityBot 43 13.5%
Google-Extended 48 15%
CCBot 59 18.5%

Download: aggregates per week (CSV) · per-domain results, latest run (CSV). robots.txt files are public; the per-domain list is there so anyone can check our numbers.

Is your own site on the list of blockers?

The free AI crawler check reads your robots.txt the same way, then requests your page with each crawler's name in the user-agent to catch firewall and CDN blocks that robots.txt does not show.

Check my site

Methodology

  • Sample: the first 500 domains of Tranco list V38GN (2026-11-17), a research ranking that combines CrUX, Cloudflare Radar, Farsight, Majestic and Cisco Umbrella. Kept in rank order, including infrastructure domains that serve no website; those mostly fail to answer and drop out of the shares.
  • Request: one HTTPS request per domain for /robots.txt only (following up to 3 redirects), user-agent SEOSetsResearchBot/1.0 (+https://seosets.com/research/ai-crawler-access), 5-second timeout, 2 seconds between domains, once a week.
  • Verdict: the group naming the crawler applies, otherwise the * group (RFC 9309). A crawler counts as blocked when that group disallows /. Path-level rules are not counted.
  • Denominator: domains whose robots.txt answered with a file or a 4xx (no file = no restrictions). Timeouts, refused connections and 5xx answers are excluded, never counted as allowing. An HTML page served at /robots.txt (a soft 404) is treated as no robots.txt.
  • Limits: robots.txt is a request, not enforcement, and a firewall or CDN can block crawlers without any robots.txt rule. This study measures stated policy only.

What each crawler does and the exact rule to allow or block it: AI crawler directory.

More research: we asked ChatGPT for the best SEO consultant in 60 cities, and which sources it cited.

Frequently asked questions

Which sites are in the study?

The Tranco top 500 domains from the Tranco research ranking (list V38GN, dated 2026-11-17), kept in rank order without hand-picking. The same list is re-checked every week, so changes over time are real changes in robots.txt, not a different sample.

What counts as blocking a crawler?

robots.txt blocks a crawler when the group that applies to it (its own User-agent group, or the * group if it has none) disallows the site root with "Disallow: /" and no "Allow: /" re-opens it. Rules that block only some paths are not counted as blocking.

Why are some domains left out of the percentages?

Shares are computed only over domains whose robots.txt answered: a file (HTTP 200) or a 4xx such as 404, which under RFC 9309 means no restrictions. Domains that timed out, refused the connection or answered 5xx are excluded rather than counted as allowing.

Does a robots.txt block mean the crawler really stays out?

No. robots.txt is a request, not an enforcement mechanism, and this study reads only robots.txt. A site can also block crawlers at the firewall or CDN, which robots.txt does not show. Use the free AI crawler check to test a page with each crawler's user-agent.

How is the data collected?

Once a week, one request per domain for /robots.txt only, with a user-agent that names SEO Sets and links to this page, a 5-second timeout and a 2-second pause between domains. No pages other than robots.txt are fetched.

SEO Sets Support

Press Enter to send · Shift+Enter for new line