G
There is an uncomfortable, open secret inside the labs of San Francisco, London, and Seattle: The AI revolution is running out of original data.
For the last seven years, the unprecedented scaling of Large Language Models followed a simple formula: throw more GPUs at more scraped text. The public internet was treated like an infinite, free aquifer of human cognition. Every forum post on Reddit, every scanned library book, every GitHub repository, and every niche hobbyist blog was pumped into massive distributed clusters to teach machines how to think.
That era is over.
We have effectively reached the bottom of the public data reservoir. Research institutes like Epoch AI have warned that the world’s supply of high-quality, human-generated public text will be completely depleted between 2026 and 2028.
There is no more pristine internet left to scrape. A great example. I texted my friend group a picture of my beer and a book. I received 3 AI generated responses that were supposed to be witty.
And right as the well is running dry, the tech industry is facing an even darker reality: The new water filling the reservoir is pure, toxic synthetic slop.
Take a snapshot of the internet today compared to 2021.
A massive, terrifying percentage of current web traffic is no longer human expression. It is programmatic SEO farms vomiting out hundreds of thousands of AI-generated affiliate articles per hour. It is LinkedIn feeds choked with synthetic thought leadership. It is AI-generated code snippets pushed to GitHub to farm metrics, AI-generated recipes padding ad pages, and automated bots arguing with other bots in the replies of X.
The internet is becoming an echo chamber of machine exhaust.
For ordinary users, this is an annoyance. For AI research labs, it is an existential crisis.
When foundation model labs spin up their crawlers to build the training sets for their next-generation models, they aren't vacuuming up organic human thought anymore. They are sweeping up the regurgitated output of GPT-4, Claude, and Llama.
Why does this matter? If an AI writes cleanly, why can’t another AI just learn from it?
Because of a mathematical inevitability known as Model Collapse.
When researchers from Oxford and Cambridge investigated what happens when generative models are trained recursively on the output of prior models, the results were catastrophic: they disintegrated.
Human language is defined by its edges: idioms, rare counter-arguments, obscure domain knowledge, eccentric metaphors, and linguistic oddities. This is the long tail of the probability distribution.
An LLM, by its very nature, is a probabilistic prediction machine that regresses toward the mean. It prioritizes the most probable, generic combinations of tokens. When you take the output of Model A and use it to train Model B, Model B loses the rare tails of the distribution.
This brings us to the great watermarking paradox.
Every software engineer knows that text watermarks whether they rely on cryptographic token-hashing or logit biasing are practically useless as an anti-cheat mechanism. Any halfway-clever high schooler can strip a watermark by prompting an open-source model to rewrite the text, translating it through two languages, or using a basic paraphrasing script.
If watermarks are so trivially broken by adversaries, why are OpenAI, Google, Anthropic, and Meta spending millions of dollars researching them?
Because the threat isn't a malicious user. The threat is the unwashed, automated internet.
When an affiliate marketer spins up an autonomous bot to publish 40,000 AI-written articles to manipulate Google Search rankings, they aren’t running expensive, multi-pass paraphrasers on every post to strip watermarks. They are making raw API calls, dumping the raw tokens into a database, and publishing it.
The tech giants don't need a watermark that defeats a motivated hacker.
Watermarking is not a copyright badge. It is a hazmat label designed to prevent their own scrapers from sucking poison into the pre-training engine.
What happens if the watermarks fail, or if open-source models without watermarks flood the internet beyond the capacity of filters to catch them?
If the public web is toxic, AI labs will stop scraping it. Instead, they are locking down private, un-polluted data vaults. We are already seeing multi-million dollar deals signed behind closed doors for Reddit archives, Stack Overflow dumps, publisher paywalls, and private messaging databases. The era of the open web being the engine of open-source AI is ending; proprietary, verified human archives are the new oil reserves.
Labs are pivoting hard to synthetic data not because they want to, but because they have to. But there is a massive distinction: they cannot use synthetic text. They have to use synthetic environments tied to mathematical or logical ground truth. A model generating 100,000 synthetic Python scripts that are subsequently executed in a sandbox compiler to prove they work doesn't cause collapse, because the compiler acts as a physical reality check. But for history, philosophy, prose, creative thought, and social nuance? There is no compiler.
If next-generation models are forced to train on a web made of 40% to 60% machine-generated slop, we will not see the leap to superintelligence. We will see the great flattening. Models will become hyper-stylized caricatures of themselves parroting the same polite, corporate, mediocre, averaged-out prose that already characterizes modern AI.
For the last 4 years people have been pushed to offload cognitive process to LLM’s. In fact I have a friend who fought a text to image LLM for over 4 hours to get a graphic just right when all he needed to do was dump the image into Adobe…… but this literally never occurred to him. So the graphic artist that was replaced by a LLM 3 years ago is going to become a hot commodity in the next few years. The writer who was told their art was dead will now cost 3 times more. Great for those who were educated pre AI. For those being educated now they are going to need to go back and re-frame what they offload to LLMs and what they need to develop and strengthen because knowing the difference will be the key for the future.
For decades, digital culture worried about what would happen when artificial intelligence became indistinguishable from human intelligence.
Almost nobody predicted the real crisis: that human intelligence on the internet would be drowned out so fast by cheap synthetic imitation that the machines would choke on their own words.
The race to detect, watermark, and isolate AI-generated content is not a philosophical debate about transparency or academic honesty. It is a desperate race against the clock.
For the last seven years, the unprecedented scaling of Large Language Models followed a simple formula: throw more GPUs at more scraped text. The public internet was treated like an infinite, free aquifer of human cognition. Every forum post on Reddit, every scanned library book, every GitHub repository, and every niche hobbyist blog was pumped into massive distributed clusters to teach machines how to think.
That era is over.
We have effectively reached the bottom of the public data reservoir. Research institutes like Epoch AI have warned that the world’s supply of high-quality, human-generated public text will be completely depleted between 2026 and 2028.
There is no more pristine internet left to scrape. A great example. I texted my friend group a picture of my beer and a book. I received 3 AI generated responses that were supposed to be witty.
And right as the well is running dry, the tech industry is facing an even darker reality: The new water filling the reservoir is pure, toxic synthetic slop.
The Great Replacement of the Open Web
Take a snapshot of the internet today compared to 2021.
A massive, terrifying percentage of current web traffic is no longer human expression. It is programmatic SEO farms vomiting out hundreds of thousands of AI-generated affiliate articles per hour. It is LinkedIn feeds choked with synthetic thought leadership. It is AI-generated code snippets pushed to GitHub to farm metrics, AI-generated recipes padding ad pages, and automated bots arguing with other bots in the replies of X.
The internet is becoming an echo chamber of machine exhaust.
For ordinary users, this is an annoyance. For AI research labs, it is an existential crisis.
When foundation model labs spin up their crawlers to build the training sets for their next-generation models, they aren't vacuuming up organic human thought anymore. They are sweeping up the regurgitated output of GPT-4, Claude, and Llama.
The Photocopy of a Photocopy: Understanding Model Collapse
Why does this matter? If an AI writes cleanly, why can’t another AI just learn from it?
Because of a mathematical inevitability known as Model Collapse.
When researchers from Oxford and Cambridge investigated what happens when generative models are trained recursively on the output of prior models, the results were catastrophic: they disintegrated.
Human language is defined by its edges: idioms, rare counter-arguments, obscure domain knowledge, eccentric metaphors, and linguistic oddities. This is the long tail of the probability distribution.
An LLM, by its very nature, is a probabilistic prediction machine that regresses toward the mean. It prioritizes the most probable, generic combinations of tokens. When you take the output of Model A and use it to train Model B, Model B loses the rare tails of the distribution.
Why Big Tech Cares About Watermarks (Hint: It’s Not About You)
This brings us to the great watermarking paradox.
Every software engineer knows that text watermarks whether they rely on cryptographic token-hashing or logit biasing are practically useless as an anti-cheat mechanism. Any halfway-clever high schooler can strip a watermark by prompting an open-source model to rewrite the text, translating it through two languages, or using a basic paraphrasing script.
If watermarks are so trivially broken by adversaries, why are OpenAI, Google, Anthropic, and Meta spending millions of dollars researching them?
Because the threat isn't a malicious user. The threat is the unwashed, automated internet.
When an affiliate marketer spins up an autonomous bot to publish 40,000 AI-written articles to manipulate Google Search rankings, they aren’t running expensive, multi-pass paraphrasers on every post to strip watermarks. They are making raw API calls, dumping the raw tokens into a database, and publishing it.
The tech giants don't need a watermark that defeats a motivated hacker.
Watermarking is not a copyright badge. It is a hazmat label designed to prevent their own scrapers from sucking poison into the pre-training engine.
The Impending Scramble: What Happens When the Slop Wins?
What happens if the watermarks fail, or if open-source models without watermarks flood the internet beyond the capacity of filters to catch them?
1. The Death of the Free, Open Web
If the public web is toxic, AI labs will stop scraping it. Instead, they are locking down private, un-polluted data vaults. We are already seeing multi-million dollar deals signed behind closed doors for Reddit archives, Stack Overflow dumps, publisher paywalls, and private messaging databases. The era of the open web being the engine of open-source AI is ending; proprietary, verified human archives are the new oil reserves.
2. Synthetic Data with Ground Truth
Labs are pivoting hard to synthetic data not because they want to, but because they have to. But there is a massive distinction: they cannot use synthetic text. They have to use synthetic environments tied to mathematical or logical ground truth. A model generating 100,000 synthetic Python scripts that are subsequently executed in a sandbox compiler to prove they work doesn't cause collapse, because the compiler acts as a physical reality check. But for history, philosophy, prose, creative thought, and social nuance? There is no compiler.
3. The Cognitive Plateau
If next-generation models are forced to train on a web made of 40% to 60% machine-generated slop, we will not see the leap to superintelligence. We will see the great flattening. Models will become hyper-stylized caricatures of themselves parroting the same polite, corporate, mediocre, averaged-out prose that already characterizes modern AI.
4. Original Human Thought Becomes More Expensive
For the last 4 years people have been pushed to offload cognitive process to LLM’s. In fact I have a friend who fought a text to image LLM for over 4 hours to get a graphic just right when all he needed to do was dump the image into Adobe…… but this literally never occurred to him. So the graphic artist that was replaced by a LLM 3 years ago is going to become a hot commodity in the next few years. The writer who was told their art was dead will now cost 3 times more. Great for those who were educated pre AI. For those being educated now they are going to need to go back and re-frame what they offload to LLMs and what they need to develop and strengthen because knowing the difference will be the key for the future.
The Bottom Line
For decades, digital culture worried about what would happen when artificial intelligence became indistinguishable from human intelligence.
Almost nobody predicted the real crisis: that human intelligence on the internet would be drowned out so fast by cheap synthetic imitation that the machines would choke on their own words.
The race to detect, watermark, and isolate AI-generated content is not a philosophical debate about transparency or academic honesty. It is a desperate race against the clock.