The internet is quietly transforming into a bot-dominated landscape. New data from TollBit shows AI bots now account for one out of every 50 website visits, a 400% surge from earlier in 2025. The finding reveals an escalating arms race between publishers desperate to protect their content and increasingly sophisticated AI scrapers that mimic human behavior to slip past defenses. With companies like Cloudflare and TollBit rushing to monetize bot access, the very fabric of the open web hangs in the balance.
The internet's demographic shift is happening faster than most realize. OpenAI, Google, and Anthropic aren't just training their AI models on web content anymore - their bots are now persistently crawling sites to fetch real-time information for chatbots and AI agents. According to fresh data from TollBit and internet infrastructure giant Akamai, this activity is reshaping the web's traffic patterns at an alarming pace.
In Q4 2025, TollBit estimates AI scrapers accounted for one out of every 50 visits to its customers' websites. Just nine months earlier, that ratio was one in 200. "The majority of the internet is going to be bot traffic in the future," Toshit Pangrahi, TollBit's cofounder and CEO, told Wired. "It's not just a copyright problem, there is a new visitor emerging on the internet."
The surge breaks down into two distinct categories. Training bots - the ones scraping content to build AI models - have been climbing steadily since July 2025, according to Akamai's data shared exclusively with Wired. But a newer phenomenon is accelerating even faster: retrieval bots that grab real-time information like product prices, movie showtimes, and breaking news to augment AI responses. Tools like ChatGPT's web browsing feature and the viral assistant OpenClaw (previously Moltbot, before that Clawdbot) are driving this second wave.
Robert Blumofe, Akamai's chief technology officer, frames the shift starkly. "AI is changing the web as we know it," he told Wired. "The ensuing arms race will determine the future look, feel, and functionality of the web, as well as the basics of doing business."
That arms race is already brutal. TollBit's report reveals that 13% of AI bot requests in Q4 2025 bypassed robots.txt files - the standard mechanism websites use to tell crawlers which pages to avoid. The share of bots ignoring these guardrails jumped 400% from Q2 to Q4 last year. Meanwhile, publisher defenses spiked 336% year-over-year as sites rushed to block unauthorized AI access.
The sophistication gap is closing fast. Some bots now disguise their traffic to look like ordinary browser requests. Others mimic human browsing patterns - varying click speeds, scrolling behavior, session lengths - making them nearly indistinguishable from real users. "The behavior of some AI agents is now almost indistinguishable from human web traffic," TollBit's study notes.
Publishers aren't taking it lying down. Condé Nast (Wired's parent company) and other media firms are currently suing several AI companies over alleged copyright infringement related to training data. But legal battles move slowly, and the technology evolves daily.
A new cottage industry is emerging to broker this tension. TollBit markets tools that let website owners charge AI scrapers for content access. Cloudflare launched a similar "pay-per-crawl" service last year. "Anyone who relies on human web traffic - starting with publishers, but basically everyone - is going to be impacted," Pangrahi said. "There needs to be a faster way to have that machine-to-machine, programmatic exchange of value."
TollBit's report identified more than 40 companies now selling web-scraping services tailored for AI applications. When Wired contacted 15 of these firms, most didn't respond. Those that did defended their practices while acknowledging the complexity.
Or Lenchner, CEO of Bright Data (one of the world's largest scraping firms), said his company's bots don't collect nonpublic information. Bright Data was previously sued by Meta and X over alleged improper scraping - Meta later dropped its suit, and a federal judge dismissed X's case.
Karolis Stasiulevičiu, a ScrapingBee spokesperson, told Wired: "ScrapingBee operates on one of the internet's core principles: that the open web is meant to be accessible. Public web pages are, by design, readable by both humans and machines."
Oxylabs, another scraping provider, argued in a statement that anti-bot systems often can't distinguish between malicious traffic and legitimate uses like cybersecurity research or investigative journalism. "The reality is that many modern anti-bot systems don't distinguish well between malicious traffic and legitimate automated access," the company said.
But not everyone's playing defense. A counterintuitive strategy called generative engine optimization (GEO) is gaining traction - essentially SEO for AI. Instead of blocking bots, companies like Brandlight help clients optimize content to appear prominently in AI tool outputs. "We're essentially seeing the rise of a new marketing channel," Uri Gafni, Brandlight's chief business officer, told Wired. "This will only intensify in 2026, and we're going to see this rollout kind of as a full-on marketing channel, with search, ads, media, and commerce converging."
The economic implications are staggering. If AI agents replace traditional web browsing for information retrieval, the entire digital advertising ecosystem - built on human eyeballs viewing ads - could collapse. Publishers lose both training compensation and real-time traffic monetization. Meanwhile, AI companies gain free access to the world's knowledge while capturing user attention in their own interfaces.
Akamai's data suggests this isn't a distant future scenario. Training bot traffic has climbed steadily for seven months. Retrieval bot activity is spiking as AI search tools proliferate. The infrastructure layer is already adapting, with major players like Cloudflare and Akamai repositioning to monetize the new reality.
What remains unclear is whether publishers can assert control before the balance tips irreversibly. The 336% surge in blocking attempts shows they're trying. But with 13% of bots already slipping past defenses, and that figure quadrupling in six months, the technical advantage may be shifting to the scrapers.
The web's transformation from human habitat to bot battleground is accelerating beyond most projections. With AI scrapers now claiming 2% of all traffic and evading defenses at quadrupling rates, publishers face an existential question: fight for control through technical and legal means, or pivot to monetization models that treat bots as paying customers. The next 12 months will likely determine whether the open web survives in recognizable form or fragments into walled gardens and paid bot access tiers. Either way, the internet that emerges won't look like the one we've known.
