The Web’s Economy Is Burning: AI Is Breaking Search

[Whitey]

Matthew Prince, CEO of Cloudflare, has issued a stark warning: AI is dismantling the economic foundation of the open web. Historically, search engines like Google operated on a reciprocal model—scraping content and directing traffic back to creators. This balance is collapsing. Today, Google may scrape six pages for every visitor it sends, while AI models like ChatGPT and Claude extract vast amounts of content with minimal or no return traffic.

This shift undermines the incentives for content creation. Without adequate recognition or revenue, creators may cease producing original content, leading to a web saturated with AI-generated material lacking authenticity and depth.

Cloudflare, observing traffic from 80% of AI companies and supporting 20–30% of the web, is uniquely positioned to address this crisis. The company is exploring solutions to realign the value exchange between content creators and AI platforms.

Some Community Questions:

How can we establish a fair compensation model for content creators in the age of AI?

What role should AI companies play in sustaining the open web ecosystem?

Can we develop standards to ensure AI tools contribute positively to content creation rather than depleting it?

The survival of the open web depends on our collective action to address these challenges.

See more: [searchengineworld.com...]

www.searchengineworld.com Cloudflare CEO Sounds the FIRE Alarm: AI Is Breaking the Web’s Economy! Search Is Broken! Ten years ago, for every two pages Google scraped, you got one visitor, today, it's six pages scraped for one visitor. And with AI? It's 250 to 1-¦ or 6,000 to 1!

[ronin]

Historically, search engines like Google operated on a reciprocal model—scraping content and directing traffic back to creators.

Haha. Ahahahahaha. Hahaha. Google in 2002?

The survival of the open web

Hmmm. This feels melodramatic.

There were various map sites before Google Maps. After Google Maps there weren't.

Generative AI may supersede / replace some elements of the currently-existing open web.

But it won't replace all, by any means.

Case in point: what generative AI could replace this web forum?

[tangor]

Case in point: what generative AI could replace this web forum?

Careful ... wouldn't be that hard! Want to bet all here has been under collection since AI came out? Want to bet that answers, originally found only here, are now in an AI quick answer at the top of the serp?

Only way to restore CONTENT CREATORS proprietary control of their intellectual property is ENFORCED copyright both domestic AND international...

Banning all AI

Another "dark web" which returns to less invasive "directories", etc. AND NO BOTS...

A sudden and miraculous conversion of HUMANITY to morality and fair play (ha!)

None of the above is likely to happen, certainly not in the foreseeable future when so many legacy websites will be shutting down for lack of traffic or ad conversions (if monetized that way) and the death of websites that host their OWN hard-earned advertising clients WHO WILL NEVER BE SEEN since the website/content will be an AI return, not a search click through...

Funny thing is that entire nations are climbing on the AI wagon building out electric grids to power it, and scrambling to gather up all the chips needed to at least get into the game but that version of AI is not quite the same copyright theft webmasters are facing at this time.

[Whitey]

Appreciate the perspectives here—some sobering, some skeptical, all valid.

@ronin – True, Google's reciprocity wasn't always a golden handshake, even back in 2002. But the *degree* of imbalance today, especially with AI scraping 250–6000 pages per visit, shifts us from friction to freefall. At least in the old model, publishers were *somewhere* in the loop. Now, they're barely even a footnote.

@tangor – You're right to point out the systemic scope. Proprietary control through enforceable copyright sounds great, but enforcement across AI LLMs, global jurisdictions, and ephemeral UGC? Herculean.

But maybe we’re looking at this backwards. What if the answer isn’t "how do we stop AI from scraping?" but rather "how do we *embed* ourselves in the AI output loop?"

If AI is the new browser, maybe the goal isn’t traffic—it’s *attribution* or *licensing* that survives the summary. A Creative Commons-style framework with mandatory payment or citation standards could be a start, if we had collective bargaining power.

[londrum]

I wouldn't mind so much if it really was an AI breakthrough, but all google is doing with their AIOs is just rewriting stuff they've scrapped. It's like a next-gen article spinner.

I remember the hoohaa when they started increasing the size of everyone's snippets to include the answer... well this is just the next version that. They've finally found a way to reprint everyone's content for free and get away with it

[tangor]

Some of the following will sound very negative. It's not. Just a dash of reality.

The problem with AI is that it is a "consensus machine" which SUMMARIZES data/content automagically. There WON'T be a single source to attribute. That creates yet another problem: Three, 300. or 30,000+ have the "answer", so who gets listed as "source"?

AI will eventually kill creators ON THE WEB. Content creators will move to other media which CAN be controlled where empires can be built and money made---any attempted theft or conversion can be dealt with in courts with lawyers and money. Go back to basics and forget the web...

Who loses? Mom and Pops, start up econ, evergreen, edu, gov, hobbies... if it can be on the web it becomes fair game. Even g admits that in those recently revealed court exhibits. You can't stop it without killing your site in current search which uses AI in any shape or form.

Sadly, I don't see a return to Ask Jeeves, Hot Point, Yahoo, AOL, and all those other dinosaur attempts to explore the web since even THAT would be exposed to all the AI bots who claim to honor robots.txt but only do so with the front facing crawler while sneaking in backdoor with "human appearing" UAs.

If one is HONEST, all want to be #1 and would do ANYTHING to get there and STAY THERE. All also think their content is the absolute best (even if it is scraped or stirred or AI generated). If there is a way to game a system to achieve superiority in making money or fame any human will do what it takes to get there, hook or crook included. Only way to combat that is law and order. AHEM, we already have way too many laws and so little order that getting anything done is a battle.

THAT SAID, I'm here for the long run (since 1996 on the www) and willing to see what happens over the next two years. I already built my web empire, made a bunch of money, kept up to six employed for near fifteen years, sold it, retired, and currently play with a hobby site and hang around since I like you guys and still find all this stuff FASCINATING. Who knows what will happen? I have a few ideas and can see some changes on the horizon, but these things are still a bit nebulous. But I'll be here for a bit longer, if the creek don't rise, and see what is coming down the pike.

[Whitey]

@londrum – Fair point. It’s hard to call it an AI breakthrough when the core output feels like polished scraping. The shift from snippet inflation to full content substitution has been gradual, but it’s a major step-change in how publishers are side-lined.

@tangor – Really value your long-term perspective. You're right that AI muddies attribution beyond practical enforcement. But rather than walking away from the web, maybe the opportunity is in rethinking how we embed ourselves into the new stack.

If AI is the front-end to discovery now, maybe we need a new layer underneath, creator metadata, verifiable attribution, even lightweight licensing baked into structured content. Sure, enforcement is another matter, but standardization might be the first step.

Just wondering aloud: What would it take to make inclusion in AI outputs conditional on attribution or compensation? Or is that already a lost cause?

[tangor]

wondering aloud: What would it take to make inclusion in AI outputs conditional on attribution or compensation? Or is that already a lost cause?

Personally think you just answered the question. Webmasters were not asked, weren't in at the beginning, and if anyone thinks about it, we weren't invited in the first place.

The concept of "the free web" is so inclusive (and invasive) that EVERYONE, even the AI folks, think that is true. Something more seismic will be required and that will come from the PAST not the "future".

IE STRONG COPYRIGHT. A means of ELECTRONIC registrations. First "found original publication". ELECTRONIC protections. Killing off theft of IP by ANYBODY---and that includes all our buddy webmasters who are NOT creators---merely those hungry for ad dollars with whatever they can copy and paste. ORIGINAL COPYRIGHT is nearly impossible to protect on the WEB, particularly since somebody can scrape AND publish your content and timestamp it 24 hours before yours. Webmasters have been complaining about this for decades and so far nothing has been done.

Computers were created to make us more creative and capable. Well, that turns out to be true. About 10% are Creative, the other 90% are copycats---or worse.

Been that way since the dawn of time and all throughout history. I'm not knocking that! After all, all human civilization is rife with different folks seeing what other different folks are doing and copying that, then expanding that, then capitalizing that, then stealing that, then going to war over that, and THAT remains what humans covet. Pretty sure the cycle will continue into Star Trek, Star Wars or even next year. :)

C'est la vie

[Whitey]

I wanted to sprinkle some hope for those transitioning towards the "new world order" whatever that means in search terms, with AI. Duane Forrester knows a thing or two about search engines having been at the coal face and being well connected (ex Microsoft). His article does a good job imo - happy reading :) [duaneforresterdecodes.substack.com...]

It’s not about ranking higher anymore, it’s about being indispensable to the answer. This is the new battleground.

duaneforresterdecodes.substack.com Search Without a Webpage How Vector DBs, Embeddings, and RRF Are Changing the Stack

[dvduval]

@tangor "AI will eventually kill creators ON THE WEB. Content creators will move to other media which CAN be controlled where empires can be built and money made"

Yes, exactly, and content creators are making videos, podcasts, and other social media content where their "humanness" is still appreciated, and not easily replaced by an LLM chat.

[tangor]

...where their "humanness" is still appreciated, and not easily replaced by an LLM chat.

Exactly. Web pages are "text", podcasts are not. THAT part would be tough for LLM (it ain't TEXT!).

On the other hand, how many webmasters---trying to make a living off the web---look good in a podcast, can "speaka my language" (nods to Men At Work), prosper, much less command any AD revenue not already held by YouTube etc.---for any kind of content? New layers of complexity!

Wanna bet AI is already doing "voice to text" LLM analysis of YouTube style video? (Merely a metaphor, not a violation of TOS---or even commonsense!)

At one time all we had to worry about is getting in the top 10. Now we worry about which 10 AI LLM scrapers WHO THEN IGNORE US.

Burning? It's a Freakin' Conflagration of Mythic Proportions!

</dr.gloom="sumboddykikamya$$">

[ronin]

content creators are making videos, podcasts and other social media content

Google's NotebookLM has been creating podcasts since September 2024 via its AI-powered Audio Overviews feature.