Applebot-Extended

It separates two uses of one crawl. Apple’s page on Applebot says the data Applebot crawls powers search in Spotlight, Siri and Safari, and may also be used to help train the foundation models behind generative AI features across Apple products. Publishers can opt out of that training by disallowing Applebot-Extended in robots.txt^[1].

The token does not crawl, and disallowing it does not remove a site from search. Apple says Applebot-Extended does not crawl webpages, that pages disallowing it can still be included in search results, and that it is used only to decide how to use data the Applebot user agent crawled. It adds that site rules for Applebot-Extended are not considered in ranking for Search^[1].

Apple states the purpose in its research posts. In June 2024 it said it trains on licensed data and on publicly available data collected by its crawler, and that publishers can opt out of the use of their web content for Apple Intelligence training with a data usage control^[2]. In June 2025 it said it follows robots.txt protocols so that publishers can opt out of their content being used to train its generative foundation models, with fine-grained controls over which pages Applebot can see and how they are used while still appearing in search results within Siri and Spotlight^[3].

A second control covers a different use. Apple says Applebot-crawled data may also give additional context and up-to-date content when AI models generate output, for example answers to broad world knowledge questions in Siri and Search that may link to sources. The opt-out for that is the nosnippet meta tag on specific content, not the token^[1]. Even with both set, Apple says, the site’s instructions may still allow Applebot to crawl, and the content stays discoverable through Spotlight, Siri and Safari^[1].

In QueryRev’s study, Applebot-Extended is blocked by name on 394 of the 6,294 sites whose robots.txt could be read, 6.3%, against 411, 6.5%, for Google-Extended^[6]. Google splits its controls differently: its AI features in Search are set by the Search generative AI control in Search Console, and Google-Extended limits the training of the models they use^[5].

In the file, host by host. Fetch robots.txt from each host the site serves and find the group that names Applebot-Extended; a crawler matches its token against the user-agent lines without regard to case^[4]. Record the paths it closes. A rule on the main domain says nothing for a subdomain with its own file.

In the logs, only Applebot appears. Because the token does not crawl, no request carries its name. Apple says Applebot traffic is generally identified by reverse DNS in the applebot.apple.com domain, or by matching the address with the CIDR ranges it publishes^[1]. Count verified requests before and after a change: they should carry on.

What Apple does with the data is not reported back to the site, so the file is the record of the choice. The part that can be observed is what the rule should leave alone: the site’s pages in Apple search results, and Applebot’s crawl rate.

Illustrative: an invented site, to show the method.

A recipe publisher wants its recipes out of Apple’s model training but still found in Siri and Spotlight. The first draft of the rule disallows Applebot for the whole site, which would end the crawl that feeds Apple search.

The rule that matches the goal names Applebot-Extended and leaves the Applebot group as it was. The publisher checks a week later that verified Applebot requests continue. A page the publisher wants kept out of Siri’s written answers gets a nosnippet tag, which is a different control for a different use.