Research · updated weekly
Which popular sites block AI crawlers?
Every week we read the robots.txt of the same 500 popular domains (Tranco top 500, a public research ranking) and record which AI crawlers each one blocks. The numbers below update on their own.
Last updated · 319 of 500 domains answered
Share of sites whose robots.txt blocks each crawler
21% (67 of 319) block at least one of the 8 AI crawlers we track for the whole site. 231 domains served a robots.txt file; the other answering domains had none (which allows everything).
| Crawler | Sites blocking | Share |
|---|---|---|
| GPTBot | 55 | 17.2% |
| OAI-SearchBot | 33 | 10.3% |
| ChatGPT-User | 35 | 11% |
| ClaudeBot | 53 | 16.6% |
| Claude-SearchBot | 36 | 11.3% |
| PerplexityBot | 43 | 13.5% |
| Google-Extended | 48 | 15% |
| CCBot | 59 | 18.5% |
Download: aggregates per week (CSV) · per-domain results, latest run (CSV). robots.txt files are public; the per-domain list is there so anyone can check our numbers.
Is your own site on the list of blockers?
The free AI crawler check reads your robots.txt the same way, then requests your page with each crawler's name in the user-agent to catch firewall and CDN blocks that robots.txt does not show.
Check my siteMethodology
- Sample: the first 500 domains of Tranco list V38GN (2026-11-17), a research ranking that combines CrUX, Cloudflare Radar, Farsight, Majestic and Cisco Umbrella. Kept in rank order, including infrastructure domains that serve no website; those mostly fail to answer and drop out of the shares.
- Request: one HTTPS request per domain for
/robots.txtonly (following up to 3 redirects), user-agentSEOSetsResearchBot/1.0 (+https://seosets.com/research/ai-crawler-access), 5-second timeout, 2 seconds between domains, once a week. - Verdict: the group naming the crawler applies, otherwise the
*group (RFC 9309). A crawler counts as blocked when that group disallows/. Path-level rules are not counted. - Denominator: domains whose robots.txt answered with a file or a 4xx (no file = no restrictions). Timeouts, refused connections and 5xx answers are excluded, never counted as allowing. An HTML page served at /robots.txt (a soft 404) is treated as no robots.txt.
- Limits: robots.txt is a request, not enforcement, and a firewall or CDN can block crawlers without any robots.txt rule. This study measures stated policy only.
What each crawler does and the exact rule to allow or block it: AI crawler directory.
More research: we asked ChatGPT for the best SEO consultant in 60 cities, and which sources it cited.
Frequently asked questions
Which sites are in the study?
The Tranco top 500 domains from the Tranco research ranking (list V38GN, dated 2026-11-17), kept in rank order without hand-picking. The same list is re-checked every week, so changes over time are real changes in robots.txt, not a different sample.
What counts as blocking a crawler?
robots.txt blocks a crawler when the group that applies to it (its own User-agent group, or the * group if it has none) disallows the site root with "Disallow: /" and no "Allow: /" re-opens it. Rules that block only some paths are not counted as blocking.
Why are some domains left out of the percentages?
Shares are computed only over domains whose robots.txt answered: a file (HTTP 200) or a 4xx such as 404, which under RFC 9309 means no restrictions. Domains that timed out, refused the connection or answered 5xx are excluded rather than counted as allowing.
Does a robots.txt block mean the crawler really stays out?
No. robots.txt is a request, not an enforcement mechanism, and this study reads only robots.txt. A site can also block crawlers at the firewall or CDN, which robots.txt does not show. Use the free AI crawler check to test a page with each crawler's user-agent.
How is the data collected?
Once a week, one request per domain for /robots.txt only, with a user-agent that names SEO Sets and links to this page, a 5-second timeout and a 2-second pause between domains. No pages other than robots.txt are fetched.