AI crawler · Common Crawl

CCBot

The crawler behind Common Crawl, an open web archive that many AI models have been trained on.

Operator
Common Crawl
Feeds
The Common Crawl open dataset
Type
Training (future models)
robots.txt token
CCBot

Full user-agent string, as Common Crawl documents it (version numbers can change):

CCBot/2.0 (https://commoncrawl.org/faq/)

What CCBot does

Common Crawl is a non-profit that maintains an open repository of web crawl data. Its dataset is widely used, including for training language models.

Does it follow robots.txt? Common Crawl tells site owners to use robots.txt to block CCBot.

How many popular sites block CCBot

18.5% of the Tranco top 500 sites whose robots.txt answered block CCBot for the whole site (59 of 319), as of . The number comes from our weekly AI crawler access study, which counts a site as blocking only when its robots.txt disallows the site root for this token.

robots.txt: allow or block CCBot

Put the group in the robots.txt at the root of every host it should cover (example.com and www.example.com each need their own).

Allow

User-agent: CCBot
Allow: /

Block

User-agent: CCBot
Disallow: /

Setting rules for several AI crawlers at once? The free AI robots.txt generator writes the groups for all of them.

A crawler follows only the most specific group that names it. Once you add a User-agent: CCBot group, it ignores your User-agent: * rules, so repeat any Disallow lines (admin areas, carts) you still want applied to it.

What blocking does and does not do

  • Does: It keeps your future pages out of new Common Crawl archives.
  • Does not: It does not remove pages from archives that were already published, or from models already trained on them.
  • Common Crawl warns that other crawlers falsely identify as CCBot, and says the real one runs on dedicated IP ranges that reverse-resolve to crawl.commoncrawl.org.

Anyone can put "CCBot" in a user-agent. To confirm a visit is genuine, compare its IP with Common Crawl's published list: https://index.commoncrawl.org/ccbot.json.

Check whether your site blocks CCBot

Enter your page and our free crawler check opens with it filled in. It reads your robots.txt for CCBot and the other nine AI crawler tokens, then requests the page as CCBot to catch a server or CDN refusal. No signup.

Frequently asked questions

How do I block CCBot in robots.txt?

Add a group that names it: "User-agent: CCBot" on one line and "Disallow: /" on the next, in the robots.txt at the root of each host you want covered.

Does blocking CCBot remove my site from AI answers?

It keeps your future pages out of new Common Crawl archives. It does not remove pages from archives that were already published, or from models already trained on them.

Is CCBot a live crawler or a training crawler?

Training: it affects what future models learn, not whether you are cited today. Does it follow robots.txt? Common Crawl tells site owners to use robots.txt to block CCBot.

How can I check whether my site blocks CCBot?

Run the free SEO Sets AI crawler check. It reads your robots.txt rules for this and nine other AI crawler tokens, and for real user agents it also requests your page with that name to see whether your server or CDN refuses it.

Official documentation

Common Crawl: CCBot. Facts on this page were read from it on November 17, 2026. Vendors change their crawlers; the vendor's page wins if it disagrees with ours.

See what to fix on your own site

The free Signal plan connects your Google Search Console and shows your top 3 pages with the most search upside, with no time limit and no credit card. AI citation tracking starts on Surge at $49 a month.

SEO Sets Support

Press Enter to send · Shift+Enter for new line