AI crawler directory
AI crawlers and how to control them
The ten AI crawler tokens our free crawler check reads, grouped by who runs them. Live crawlers fetch pages to answer users now, which is how you get cited. Training crawlers collect pages for future models. Blocking one kind does not block the other. Facts are from each vendor's own documentation, checked November 17, 2026.
OpenAI
Anthropic
Perplexity
Common Crawl
Is any of these blocked on your site?
Our free check reads your robots.txt for every crawler above and, for the real user agents, requests your page with each name to catch a firewall or CDN that refuses them. No signup.
Run the AI crawler checkReady to set your own rules? The free AI robots.txt generator writes allow or block groups for every crawler on this page.
How many popular sites block these crawlers? See our weekly AI crawler access study of the Tranco top 500. Block shares on this page are from its latest run, finished November 18, 2026.
Related
- AI crawler checkFree, no signup: is any AI crawler blocked on your site?
- AI robots.txt generatorFree: allow or block each AI crawler and copy the rules.
- AI crawler directoryEvery AI crawler our check reads, with robots.txt rules.
- Claim your business in ChatGPTA first-hand, step-by-step guide.
- Compare SEO SetsDated, sourced comparisons with Otterly and Peec.
See what to fix on your own site
The free Signal plan connects your Google Search Console and shows your top 3 pages with the most search upside, with no time limit and no credit card. AI citation tracking starts on Surge at $49 a month.