AI crawler directory

AI crawlers and how to control them

The ten AI crawler tokens our free crawler check reads, grouped by who runs them. Live crawlers fetch pages to answer users now, which is how you get cited. Training crawlers collect pages for future models. Blocking one kind does not block the other. Facts are from each vendor's own documentation, checked November 17, 2026.

OpenAI

Anthropic

Perplexity

Google

Common Crawl

Is any of these blocked on your site?

Our free check reads your robots.txt for every crawler above and, for the real user agents, requests your page with each name to catch a firewall or CDN that refuses them. No signup.

Run the AI crawler check

Ready to set your own rules? The free AI robots.txt generator writes allow or block groups for every crawler on this page.

How many popular sites block these crawlers? See our weekly AI crawler access study of the Tranco top 500. Block shares on this page are from its latest run, finished November 18, 2026.

See what to fix on your own site

The free Signal plan connects your Google Search Console and shows your top 3 pages with the most search upside, with no time limit and no credit card. AI citation tracking starts on Surge at $49 a month.

SEO Sets Support

Press Enter to send · Shift+Enter for new line