Meta’s New Terms Could Let It Scrape News Sites for Free

Overview

Meta is set to introduce sweeping new commercial terms on October 30th that could fundamentally alter how the social media giant interacts with online publishers. According to a report by Nieman Lab, the updated policy grants Meta permission to crawl and scrape websites “associated” with business accounts across its platforms, including Facebook, Instagram, WhatsApp, and Messenger. This expansion appears to give the company broad access to external content for the purpose of developing, improving, and training its artificial intelligence products.

The move has sparked significant concern among legal experts and news organizations, who warn that the new terms may allow Meta to bypass traditional licensing agreements and robots.txt restrictions. While Meta maintains that it is merely consolidating existing commercial and self-serve ad terms, critics argue the language effectively gives the company carte blanche to utilize publisher content for AI training without compensation.

The Scope of the New Policy

The updated terms apply to any business account that promotes its products or services across Meta’s suite of applications. The language explicitly authorizes Meta and its agents to use automated software—including bots, spiders, crawlers, and user agents—to access, index, cache, and collect data from domains associated with those businesses.

“You authorize us... to use automated software... to access, index, cache, analyze, collect, retrieve, or crawl the domains, web pages, or URLs... and to periodically pull updates from such domains, web pages, or URLs.”

This authorization extends to any content made available through the use of Meta’s products by either the business or its users. Crucially, the terms state that Meta may use any content scraped from these associated domains to “provide, develop, improve, research, train, and maintain the integrity of” its commercial products and services.

The breadth of this definition is what has raised alarms. For many online publishers, their website is intrinsically linked to their social media presence. By promoting content on Facebook or Instagram, a news outlet may inadvertently classify its entire website as an “associated” domain, thereby granting Meta free rein to scrape its archives and recent publications.

Legal Implications and robots.txt Conflicts

The central point of contention lies in the potential conflict between Meta’s new terms and the robots.txt protocol. The robots.txt standard is a widely accepted convention that allows website owners to instruct web crawlers which parts of their site should not be accessed. It serves as a digital boundary marker for content ownership and privacy.

Legal experts consulted by Nieman Lab suggest that Meta’s new commercial terms could be interpreted as superseding these blocking instructions. If a business agrees to Meta’s updated terms, they are granting Meta explicit permission to crawl their associated websites. Experts argue Meta could use this contractual agreement to assert that its commercial rights override a site owner’s technical preference to block crawlers via robots.txt.

James Grimmelmann, a professor of digital and information law at Cornell University, highlighted the severity of this shift. He noted that for news publishers, the policy clearly represents permission for Meta to train AI on any content from their sites.

“For news publishers, this clearly represents permission to Meta to train AI on any content from their sites and possibly to deploy some of that content, even in competition with the publishers themselves.”

This interpretation suggests that Meta might no longer need to negotiate individual licensing deals with every publisher whose content it wishes to use for AI model training. Instead, it could rely on the blanket authorization granted through its commercial terms of service.

Meta’s Defense and Strategic Context

In response to the reporting, Meta spokesperson Daniel Roberts issued a statement denying that the company is changing its approach to web crawling. He asserted that there is no change to how Meta uses web crawlers and characterized suggestions to the contrary as false. Roberts emphasized that content owners can still use robots.txt flags to indicate their preferences, pointing to Meta’s developer documentation as evidence of their adherence to standard web protocols.

“There is no change to how Meta uses web crawlers; any suggestion to the contrary is false… Content owners can still use robots.txt flags to indicate their preferences, as we clearly explain in our developer documentation.”

However, the timing and context of this policy change raise questions about Meta’s broader strategy. Last month, Meta launched a new subscription service called Meta One, which requires businesses and creators to pay in order to include links in their posts and reels. This creates a paradoxical situation where Meta is charging users for the ability to share links to specific content, while simultaneously preparing to crawl and scrape that same content for free to train its AI.

The company has previously pursued direct licensing deals with major news organizations. Meta has struck AI licensing agreements with entities such as CNN, Fox News, USA Today, and People Inc. These deals involve direct compensation to the publishers. The new terms, however, could allow Meta to access a much broader swath of content from smaller publishers and independent sites that have not entered into such agreements.

Impact on Publishers and the AI Industry

For the publishing industry, the implications are profound. Many outlets rely on social media traffic to drive readership to their websites. If promoting content on Meta’s platforms grants the company ownership-like rights to that content’s digital footprint, publishers may face a difficult choice: opt out of Meta’s commercial tools entirely or concede access to their intellectual property.

The potential for Meta to deploy scraped content in competition with the very publishers it relies on for engagement is a significant fear. As AI-generated summaries and content replicas become more prevalent, publishers worry that their original reporting could be used to train models that eventually serve as substitutes for their own products.

Furthermore, the “take it or leave it” nature of the terms poses a challenge to smaller businesses. The policy states that businesses that do not agree to the new policies “may stop all access or use of our Products for business or commercial purposes.” Given Meta’s dominance in digital advertising, many businesses feel they have little choice but to comply, even if they disagree with the data usage provisions.

As the October 30th implementation date approaches, the industry awaits further clarification from Meta and potential legal challenges. The outcome of this policy shift could set a precedent for how large tech platforms interact with the open web and how AI training data rights are defined in the commercial sector.