In a recent analysis conducted by data journalist Ben Welsh, it was discovered that a significant number of news websites are blocking certain AI bots from crawling their sites. This raises questions about the reasons behind this blocking and the implications for both news publishers and AI developers.
Welsh’s analysis revealed that 27% of the news websites surveyed are blocking Applebot-Extended, while a staggering 53% are blocking OpenAI’s bot. Additionally, Google’s AI-specific bot, Google-Extended, is being blocked by 43% of the sites. These numbers indicate a significant trend in news publishers taking measures to block AI crawlers from accessing their content.
According to Welsh, there seems to be a divide among news publishers regarding their decision to block AI bots. Some speculate that licensing deals and partnerships may play a role in these decisions. For instance, The New York Times reported that Apple was in talks with publishers for AI deals, while other competitors like OpenAI and Perplexity have announced partnerships with various news outlets and platforms. This suggests that there may be business strategies and incentives involved in allowing or blocking AI bots.
One of the challenges faced by news publishers is the constant emergence of new AI agents and the need to manually edit the robots.txt file to block them. This task can be overwhelming, particularly with the increasing number of AI crawlers entering the scene. Without proper management, publishers may struggle to keep up with the ever-changing landscape of AI bots.
The blocking of AI crawlers has become a top priority for media executives, with some CEOs directly involved in deciding which bots to block. Some outlets have explicitly stated that they block AI scraping tools unless they have a commercial agreement with the owners. This highlights the importance of partnerships and agreements between news publishers and AI developers in navigating the digital landscape.
The trend of news websites blocking AI crawlers signifies a shift in how publishers approach content access and data sharing. With the rise of AI technology, it is essential for news outlets to carefully consider the implications of blocking or allowing AI bots to crawl their sites. Moving forward, collaboration and partnerships between publishers and AI developers will play a crucial role in shaping the future of digital content delivery.
Leave a Reply