Discover how to keep your content discoverable while preventing AI training with new settings from Cloudflare.
Website owners have long faced a daunting dilemma: How do you remain visible in search engines while safeguarding your content from being used for AI training? This challenge has stemmed from the use of mixed-use crawlers by major microsoft-and-teachers-union-strike-ai-privacy-agreement/">tech companies, which often combine functionalities for both search and AI purposes. The latest announcement from ai-driven-enterprises/">Cloudflare introduces new controls that promise to bridge this gap, giving owners more agency over their content.
Traditionally, the web has operated under a straightforward principle: the more discoverable you are in search results, the more traffic you gain. For most website owners, search is essential — less than 1% of sites managed by Cloudflare actively choose to block search bots. However, allowing your content to be indexed for search could inadvertently lead to its usage for AI training, which some website owners find unacceptable.
A significant portion of web revenue comes from advertisements or subscriptions that depend on actual human engagement. If an AI model trains on your content instead of a person visiting your site, it could affect your revenue stream. This is why more granular control over crawling behaviors is increasingly necessary.
In a bid to address these conflicts, Cloudflare has rolled out a new Disallow AI Training setting. This revolutionary feature enables website owners to remain indexed on search engines while explicitly denying their content to be used in AI training.
Under this update, major crawler operators like Apple, Google, and Microsoft have committed to respecting these settings. This is a significant step forward because it separates the functionalities of web crawlers, allowing content owners to selectively disallow AI training.
The Disallow AI Training setting utilizes a directive in the robots.txt file, offering a much clearer method of signaling preferences. The previous distinction between “Block AI” and “Block on pages with ads” did not apply to mixed-use crawlers due to the potential loss of discoverability. However, this new setting allows for the simultaneous engagement in search results while blocking AI training.
With the launch of this setting, Cloudflare is not only enhancing its current bot management capabilities but is also pushing for more transparent practices across the board. The intention is to empower site owners with the knowledge of what crawlers are doing with their content and to give them the tools to make informed decisions.
The introduction of an Accountable designation for crawler operators serves to identify those that adhere to these new protocols. Apple, Google, and Microsoft meet the Accountable criteria, proving their commitment to respecting publisher choices and maintaining transparency around how crawlers use the published content.
Each organization has its measures in place to allow site owners to opt out of AI training while retaining their visibility in search results. For example, Applebot offers the option to add a Disallow rule for its extended crawlers, while Googlebot provides additional tools and metrics to help site owners manage their preferences effectively.
Despite this advancement, challenges remain regarding how AI summaries are managed. AI summaries can impact user engagement significantly by providing answers directly, potentially diminishing website visits. While more than 50% of consumers read AI summaries, they are notably 40% more likely to conclude their search after encountering one.
However, AI can also drive higher conversion rates. Users referred by AI search often convert at a rate three to five times greater than those coming from traditional search. For businesses focused on sales, fewer visits with a higher intent may be more beneficial than a higher volume of less engaged visitors.
Cloudflare's nascent efforts to allow site owners more control over how much of their content is included in AI summaries illustrate the progressive journey toward giving publishers a voice in how their information is presented and used.
The new settings and controls from Cloudflare mark a significant step in achieving a balance between the necessity of website discoverability and the desire to protect content from misuse in AI training. Collaboration between infrastructure providers, content creators, and technology companies must continue to ensure solid standards and practices are in place.
With ongoing discussions about how AI impacts publishers, the importance of adopting standards like ai-prefs will play a critical role in optimizing how contents interact with AI systems in the future. The path towards a fair balance for all stakeholders in the digital space is being navigated with each progressive step made today.
Website owners are encouraged to adopt these new controls available on all Cloudflare plans. This proactive approach is necessary to maintain control over their digital assets without sacrificing visibility in search results.
The Disallow AI Training setting works by publishing a preference in the robots.txt file, allowing crawlers to know that website owners do not permit their content to be used for AI training while still being indexed for search.
No, using the Disallow AI Training setting will not negatively impact your site's search ranking. Both Apple and Google have confirmed that opting out of AI training does not interfere with how your site is indexed in search results.
Website owners can expect future enhancements regarding how they manage their content in relation to AI summaries. Cloudflare aims to provide more granular controls, allowing site owners to decide how much of their content appears in AI-generated excerpts.