A coalition of major web publishers just fired the opening shot in what could become the industry's biggest battle over AI training data. Reddit, Yahoo, Medium, Quora, and People Inc. announced support for the Really Simple Licensing (RSL) Standard - a new framework that lets publishers set exact pricing terms for AI companies scraping their content. The timing isn't coincidental: AI companies are burning through billions training next-generation models, and publishers want their cut.
The web's latest power play just dropped, and it's aimed squarely at AI companies who've been feasting on free content for years. Reddit, Yahoo, Medium, Quora, and People Inc. are banding together behind the Really Simple Licensing (RSL) Standard, a framework that transforms the humble robots.txt file into a pricing menu for AI training data.
The move represents the most coordinated publisher response yet to AI companies' massive data appetite. Where individual negotiations have yielded mixed results - The New York Times and News Corp secured deals with OpenAI, while others got nothing - the RSL Standard promises collective bargaining power.
"The goal is to create a new, scalable business model for the web," RSL Collective co-founder Eckart Walther told The Verge. Walther, who helped create RSS, is betting that unified publisher action can force AI companies to pay up. "RSL takes some of those early RSS ideas and creates a new layer for the entire internet where licensing rights and compensation rights are defined."
The technical implementation builds on the existing robots.txt protocol that's governed web crawling since the 1990s. But instead of simple allow/deny instructions, publishers can now embed licensing terms directly in their robots.txt files. The system supports multiple pricing models: subscription fees, pay-per-crawl charges, and even pay-per-inference fees that compensate publishers each time an AI model references their content in responses.
This isn't just about blocking bots - it's about monetizing them. Publishers using the RSL Standard can set different rates for different types of crawling. Search engine bots and archival services can proceed as usual, while AI training crawlers face pricing walls.
Behind the scenes, the RSL Collective is working with Fastly, a major content delivery network, to enforce these licensing terms. "Fastly is the bouncer at the door to the club, and they won't let people in unless they have the right ID," explains Doug Leeds, the collective's other co-founder and former CEO of IAC Publishing. "RSL is issuing the IDs."
