AI Search
CCBot
CCBot is the web crawler of Common Crawl, a non-profit that publishes an open archive of web pages that anyone may download and analyse, and from which language models have been trained. CCBot obeys robots.txt, so a Disallow rule in a group for CCBot stops future crawls of the site.
- Industry term
By QueryRev ResearchPublished Updated6 min read
Why it matters.
Common Crawl is not an AI company. Its FAQ calls it a 501(c)(3) non-profit that provides a copy of the Internet to researchers, companies and individuals at no cost, and says its dataset is a sample of the web: it does not generally archive an entire website, but a randomly selected subset of it[2]. A rule for CCBot is therefore a choice about an archive that others download, not about one company’s product.
The archive’s reach is the reason sites write the rule. In a 2020 paper on a large language model, filtered Common Crawl data was 60% of the weight in the model’s training mix, 410 billion tokens and the largest single part[4]. That describes one model in 2020. Common Crawl does not train models itself, and nothing here says which models use which crawl today. Common Crawl says each month it crawls several billion new pages, and that its datasets have been cited in over 10,000 research papers[3].
The crawler identifies itself as CCBot/2.0 and checks robots.txt first, fetching a page only if crawling it is allowed. It does not execute JavaScript and does not use cookies[2]. Common Crawl says it runs on dedicated address ranges with reverse DNS under crawl.commoncrawl.org, and warns that crawlers falsely identify themselves as CCBot, so user agent strings should be verified[1].
The controls are the ones of the Robots Exclusion Protocol, which RFC 9309 sets out. A crawler finds its group by matching its product token against the user-agent lines without regard to case, and a crawler with no group of its own follows the catch-all group[6]. Common Crawl documents two rules: Disallow: / under User-agent: CCBot stops the crawl of the site, and a Crawl-delay value slows it; it says it obeys the parameter and checks the file again from time to time[2].
Many sites refuse it. In QueryRev’s study, CCBot is blocked by name on 594 of the 6,294 sites whose robots.txt could be read, 9.4%, second only to Bytespider, on 595, 9.5%[5].
How it is measured.
In the file, host by host. Fetch robots.txt from each host the site serves and find the group that applies to CCBot: a group naming it, or the catch-all group when there is none. Record what that group closes. A rule on the main domain says nothing for a subdomain with its own file.
In the logs, by address. Filter requests whose user agent contains CCBot, then verify each one: the address should resolve to a name under crawl.commoncrawl.org, and that name should resolve back to the same address[1]. A request that fails the check is not CCBot, whatever its user agent says. Common Crawl also publishes its address ranges as JSON.
Around a change. Date the edit to the file, then count verified CCBot requests per day for the weeks either side. They should stop after the crawler has read the new file. A site that still sees verified requests weeks later should check that the file it edited is the one served on that host.
Measured by
- Rule per host
- Verified requests
- Requests after a rule change
A worked example.
Illustrative: an invented site, to show the method.
A regional newspaper sees in its logs that a crawler named CCBot fetches thousands of pages a day. Its robots.txt has a catch-all group that allows everything and no group for CCBot. The editors want their archive left out of open datasets and have no quarrel with search engines.
They add a group for CCBot that disallows the whole site and leave every other group as it was. After the crawler reads the new file, the requests that verify under crawl.commoncrawl.org stop. A group holding only Crawl-delay: 2 would have slowed the crawler to one request every two seconds, and would not have stopped it.
What to do, step by step.
Decide whether the site belongs in an open web archive. Common Crawl serves researchers as well as model builders, so the choice is not only about AI.
To keep CCBot out, add a group: User-agent: CCBot with Disallow: /. To keep it from some paths only, disallow those paths.
To slow it without stopping it, set Crawl-delay for CCBot. Common Crawl says it obeys the parameter[2].
Put the group in the robots.txt of every host the site serves, and check that the file answers 200.
Verify addresses before you act on logs: reverse DNS under crawl.commoncrawl.org, then forward DNS back to the address[1]. Deal with fakes at the firewall, because a robots.txt rule only reaches a crawler that reads it.
For a legal request about content already in the archive, use the route Common Crawl documents; it lists such requests in its Opt-Out Ledger[3].
Mistakes to avoid.
Believing a rule removes what is already archived. A Disallow rule governs what CCBot fetches from then on. Common Crawl handles legal opt-out requests separately and publishes them as an Opt-Out Ledger[3]. A robots.txt change is not a deletion of past crawls.
Trusting the user agent string. Common Crawl says other crawlers pose as CCBot, and gives reverse DNS as the check[1]. A log line that names CCBot proves nothing alone.
Writing Crawl-delay and expecting a block. Crawl-delay asks CCBot to wait between requests[2]. It does not stop the crawl. The block is Disallow: /.
Expecting a rule to leave AI search. CCBot belongs to an archive, not to an AI search product. The tokens of those products are separate groups with their own documentation, so read each vendor’s page for the token it names.
Treating it as the only route into training data. Filtered Common Crawl was 60% of the training mix of that 2020 model, and the rest came from other datasets[4]. A CCBot rule is one control, and it says nothing about data a model builder gets elsewhere.
How QueryRev treats it.
QueryRev reads a site’s robots.txt host by host and shows the group that applies to CCBot beside those of the other AI crawlers, so a choice about an open archive is not mistaken for one about an AI product. A change to the file is a change to the site: it passes your policy and is written to the Change Ledger with the file before and after. See the guide to GPTBot and robots.txt and the entries on AI crawlers and user agents.
Questions people ask.
Does CCBot obey robots.txt?
Yes, according to Common Crawl. It checks robots.txt first and fetches a page only if crawling it is allowed. A group for CCBot with Disallow: / stops the crawl, and a Crawl-delay value slows it. A rule applies to future crawls, not to pages already in the archive.
Is CCBot an AI crawler?
It is a crawler for an open archive, and that archive has been used to train language models. Common Crawl does not train models itself. QueryRev’s study counts it among the AI crawlers it measures because of that use, and a site decides about it as it decides about the rest.
How do I tell the real CCBot from an impostor?
Check the address. Common Crawl runs CCBot on dedicated ranges with reverse DNS under crawl.commoncrawl.org: the address should resolve to a name under that domain, and the name should resolve back to the address. It also publishes the ranges as JSON.
Sources and citation.
Primary sources only: the search engines, the AI vendors, standards bodies and QueryRev’s own research. Each is cited in the text by its number.
CCBotCommon Crawl ·
Common Crawl FAQCommon Crawl ·
Common Crawl Foundation Opt-Out Registry (17 September 2025)Common Crawl ·
Language Models are Few-Shot Learners (Brown et al., 2020)arXiv ·
State of AI crawler access, October 2026QueryRev Research ·
Cite this definition
A suggested citation. Its link goes to the definition, whose anchor does not change.
QueryRev Research. “CCBot.” QueryRev AI glossary, 24 October 2026. https://queryrev.com/glossary/ccbot#definition
Related terms