TL;DR:
• Reddit blocks Internet Archive from indexing posts, comments, and profiles after detecting AI scraping
• Only Reddit homepage can be archived going forward, limiting historical preservation
• Part of Reddit's broader data monetization strategy following Google and OpenAI deals
• Anthropic lawsuit shows Reddit's aggressive stance on unauthorized AI training
Reddit just pulled the plug on the Internet Archive's ability to preserve its content, claiming AI companies have been using the Wayback Machine as a backdoor to scrape user data. Starting today, the digital preservation service can only archive Reddit's homepage – effectively wiping out years of conversation history from the public record.
Reddit is cutting off one of the internet's most important digital preservation tools in its escalating war against unauthorized AI training. The company detected AI companies scraping its data through the Internet Archive's Wayback Machine and responded by severely restricting what the service can preserve.
"Internet Archive provides a service to the open web, but we've been made aware of instances where AI companies violate platform policies, including ours, and scrape data from the Wayback Machine," Reddit spokesperson Tim Rathschmidt told The Verge. The restrictions went into effect today after Reddit gave the Archive advance notice.
The move effectively erases Reddit from the historical internet record. Where researchers, journalists, and curious users could once browse deleted posts and track how conversations evolved over time, they'll now find only snapshots of which topics were trending on Reddit's front page. The decision impacts millions of posts, comments, and user profiles that previously lived in the Archive's vast digital library.
[embedded image: Reddit's data protection measures visualization]
This latest restriction fits Reddit's established pattern of monetizing data access while aggressively blocking unauthorized scraping. The company struck lucrative deals with both Google and OpenAI for AI training data, while simultaneously cutting off access for companies unwilling to pay. Last year's infamous API changes – which sparked massive user protests and forced popular third-party apps like Apollo to shut down – followed the same playbook.
