Publishers Can Reject Apple’s AI Training Without Leaving Search
Apple continues to reserve the option to use publicly available web content for foundation-model training, but publishers can refuse that use without removing their pages from Apple’s search products. Blocking Applebot-Extended affects training permission rather than ordinary Applebot crawling.
That distinction sharpens the meaning of the publisher backlash first documented in 2024. Prominent media companies and online platforms did opt out, but the durable issue is not whether websites must block Apple entirely: it is whether creators can preserve discovery while setting different terms for model training and AI-generated answers.
The 2024 resistance was concentrated among publishers
In WIRED’s August 2024 investigation, Facebook, Instagram, Craigslist, Tumblr, The New York Times, the Financial Times, The Atlantic, Vox Media, the USA Today network and Condé Nast were among the organizations excluding content from Apple’s AI training; two samples of 1,000 high-traffic websites put the blocking rate at roughly 6% and 7%, while 294 of 1,167 primarily English-language US news sites blocked Applebot-Extended.
The figures did not support a web-wide rejection of Apple. They instead showed a marked divide between the broader web and news organizations, whose articles, archives and subscription businesses gave them stronger reasons to scrutinize automated reuse.
Those findings remain a historical snapshot rather than a current directory. Any organization can revise its robots.txt file after a policy change, commercial agreement or technical migration, so a block observed in 2024 does not prove that the same rule is active today. What endured was the underlying distribution conflict: publishers may value referrals from Apple products while withholding permission to use the same material for model development.
Apple separates crawling, training and generated answers
Apple’s June 2026 Applebot documentation distinguishes ordinary search crawling from foundation-model training and the use of current web material as context for AI-generated output. It also states that Applebot-Extended does not crawl pages itself; it controls how data already collected by Applebot may be used.
The framework creates three separate decisions for a publisher:
- Search crawling: directives aimed at Applebot determine whether it may crawl and index pages for experiences including Spotlight, Siri and Safari.
- Foundation-model training: directives aimed at Applebot-Extended determine whether Applebot-collected content may be used to train Apple’s general-purpose foundation models.
- AI answer context: a page-level nosnippet directive prevents the content from being used as additional, current context for generated output and limits eligible suggestions to the page title.
A site can therefore disallow Applebot-Extended while continuing to permit Applebot. Its pages may remain eligible for search results even though their content is excluded from the training use governed by the Extended directive. Preventing crawling altogether requires rules directed at Applebot itself.
Paywalled material has another distinct control. A page marked with the schema.org property isAccessibleForFree: false can remain eligible for search results, but its content is excluded from the additional context supplied to AI models generating output for Apple products and services. The signal works at page level and does not replace Applebot-Extended when the objective is to opt out of foundation-model training.
This separation matters to subscription publications because search visibility, answer generation and training are economically different uses. A publisher can preserve a route to its paywall without automatically allowing the full article to become current context for a generated response.
Managed opt-outs are easier, but robots.txt is not enforcement
Cloudflare’s managed robots.txt documentation, updated in August 2026, includes a disallow rule for Applebot-Extended and an ai-train=no preference while leaving search permitted; the managed feature is available across its plans and can be combined with an existing robots.txt file.
This lowers the operational burden for independent creators and small publications that do not maintain crawler lists themselves. A newsletter archive, portfolio or membership site can express a training preference without using the same rule to close every route to discovery.
However, robots.txt communicates instructions rather than forming a technical barrier. Compliance is voluntary, and the file cannot stop a crawler from requesting publicly accessible content. Enforced blocking requires a traffic-control layer that can identify and reject requests at the server or network edge.
Applebot is designed to observe the relevant standard robots.txt directives during general search crawls. Even so, publishers should distinguish a declared preference from an access control: one tells a cooperating crawler what use is permitted, while the other actively prevents delivery of the page.
The decision is about distribution rights, not one master switch
The current framework turns a blunt choice into a set of content-distribution decisions. Allowing Applebot can preserve eligibility for discovery, disallowing Applebot-Extended can withhold material from the training use it governs, and nosnippet can separately restrict descriptions, web answers and current AI-answer context.
The appropriate combination depends on the publication’s business model. A subscription publisher may place greater value on keeping full text outside generated answers, while a creator seeking reach may permit snippets and indexing but reject foundation-model training. A publisher negotiating content licenses may view training access as a right that should be granted through an agreement rather than by default.
None of these directives transfers copyright, creates a licensing contract or settles whether a particular use is legally permissible. Their practical value is narrower: they let creators state different preferences for crawling, training and generated output without treating disappearance from search as the unavoidable price of refusing AI training.
Also read:
Subscribe to our newsletter
Get the latest Web3, AI, and crypto news delivered straight to your inbox.