The ouroboros problem

“The serpent that eats its own tail.” — Ancient symbol

The obvious response to the data wall is to generate more data. If human-produced text is finite, perhaps model-produced text can fill the gap. This idea has a name in the industry: synthetic data augmentation. It also has a problem.

Model collapse

Shumailov and colleagues published the definitive analysis in Nature in 2024. They showed that language models trained on data generated by other language models (or recursively by themselves) undergo model collapse: a progressive degradation of the output distribution that unfolds in two phases.

In the early phase of collapse, the distribution tails disappear. Rare events, minority patterns, unusual phrasings, low-frequency but genuine features of the original distribution, are the first casualties. The model converges toward the mode of the distribution, losing the diversity that characterized the original data. In practical terms: the outputs become more generic, more repetitive, more “average.”

In the late phase, the model loses most of its variance entirely, converging toward something approaching a delta function: a distribution concentrated on a single output pattern. By this stage, generated text is incoherent and repetitive.

The quantitative trajectory is striking. Shumailov and colleagues measured perplexity degradation across generations of recursive training. By generation 4 to 6, perplexity has degraded by 60 to 80 percent. The outputs are recognizably collapsed. The process is not gradual in the way that might allow careful monitoring and correction; it accelerates as each generation’s training data is further from the original distribution.

Why collapse happens

The mechanism is iterated lossy compression. A generative model is an imperfect estimator of its training distribution. It assigns slightly too much probability to common patterns and slightly too little to rare ones. When the next model trains on this slightly biased output, the bias compounds. Common patterns become more common; rare patterns become rarer. Across multiple generations, the rare patterns vanish entirely. This is not a bug in any particular model architecture. It is a mathematical consequence of iterating any imperfect estimator: each application of the map pushes the distribution toward lower entropy.

The ouroboros, the serpent eating its own tail, is an apt metaphor. A system that feeds on its own output converges to a distribution with lower entropy than its input. The information lost at each step is not recoverable from the output alone.

The mixing solution

Gerstgrasser and colleagues showed in 2024 that collapse can be avoided if the original human-generated data is preserved and mixed with synthetic data at each generation. This is a genuine finding and should be credited honestly. But it addresses a different problem than the one scaling needs to solve. Preserving original data alongside synthetic data prevents regression below the original model’s performance. It does not extend the frontier. The model trained on a mixture of real and synthetic data does not outperform the model trained on real data alone, because the synthetic data contains no information that was not already in the original model. Synthetic data can regularize, can provide augmented views of existing patterns, can improve robustness. It cannot create new knowledge.

The information conservation law

The data wall is not a software problem awaiting an engineering solution. It is a conservation law: you cannot extract more information from a corpus than the corpus contains, and you cannot extend a corpus by generating text from a model trained on that corpus. The information is already inside the model. Writing it out and reading it back in does not create more of it.

Data decay in an AI-saturated internet

There is a second, more insidious problem. As AI-generated content proliferates across the internet, the quality of future training data degrades. Estimates suggest that by 2026, the majority of web content may be AI-generated. If future models scrape this AI-saturated web, they will be training on a mixture of human and synthetic data, unintentionally ingesting model collapse into their training pipeline.

The snake begins eating its tail not by design, but by accident. The commons is polluted. High-quality human-generated text, the irreplaceable substrate of language model training, becomes increasingly difficult to separate from synthetic imitations. This is not a hypothetical future risk. It is already happening.

Data decay: the evidence

The AI pollution of the internet is not speculation. It is measurable, observable, and accelerating.

Stack Overflow: In late 2022, Stack Overflow banned ChatGPT-generated answers after moderators observed a flood of plausible-sounding but often incorrect responses. Analysis by the community found that AI-generated answers had higher rates of subtle errors, misleading explanations, and hallucinated references. Despite the ban, enforcement is difficult: distinguishing AI-generated text from human-written text is non-trivial, and determined users continue posting AI-generated content. By mid-2023, estimates suggested 10-30% of new answers contained AI-generated components. The signal-to-noise ratio is degrading.

Wikipedia: Wikipedia editors have engaged in ongoing battles over AI-generated content. The English Wikipedia’s policy prohibits submitting AI-generated text without human verification, but enforcement relies on volunteer moderators detecting subtle tells. Multiple studies have found AI-generated Wikipedia edits slipping through: articles created by LLMs, biographies with hallucinated details, citations to non-existent papers. The problem is worse in smaller language Wikipedias with fewer active moderators. Wikipedia’s quality as a training corpus is declining.

Academic preprint servers: ArXiv, bioRxiv, and other preprint servers have seen a surge in papers with AI-generated sections or entirely AI-generated content. These range from papers using ChatGPT to write summaries (which may be acceptable) to entirely synthetic papers with fabricated results (which are not). Detection is difficult: modern LLMs generate grammatically correct, stylistically appropriate text that passes superficial review. Several high-profile retractions have occurred after peer review caught fabricated data in AI-generated papers, but many likely slip through.

News and content farms: Low-quality news sites and content farms have adopted AI generation at scale. Some sites publish hundreds of AI-generated articles daily, optimized for SEO but providing minimal information value. Google’s search index is increasingly contaminated with this content. While Google’s algorithms attempt to penalize low-quality content, the arms race between AI generation and detection favors generation. The median quality of web text is declining.

Social media: Twitter, Reddit, and other platforms report surges in bot activity using LLM-generated text. These bots engage in conversations, post comments, and generate content that appears human. Detection is difficult at scale. Reddit’s r/SubSimulatorGPT2 demonstrates how convincing AI-generated posts can be; distinguishing them from human posts requires careful attention. As LLMs improve, the distinction becomes harder.

Code repositories: GitHub Copilot and competitors have led to a surge in AI-assisted code. While much of this code is functional, it also propagates common bugs, deprecated patterns, and security vulnerabilities that the model learned from training data. Code review catches some of this, but much is committed. Future models training on this code will learn not only from human-written code but from AI-generated code that may contain systematic errors.

Quantitative estimates: A 2023 study by researchers at AWS AI Labs estimated that by 2025, 50-90% of new text on the internet may be AI-generated or AI-assisted, depending on the domain. News articles, social media posts, and blog content are highest; academic papers and books are lowest (but still significant). By 2027, the median web page scraped for training data may be majority AI-generated.

This is not a hypothetical scenario in which future models might encounter data pollution. It is happening now. Models trained in 2025 and beyond will ingest this polluted data unless extraordinary effort is made to filter it out. But filtering is difficult: AI detection tools have false positive rates of 5-20%, meaning that aggressive filtering discards significant amounts of genuine human-generated content along with the AI-generated noise.

The ouroboros has begun. The snake is eating its tail. The commons is degraded.

Where synthetic data actually works

The picture painted so far is grim: synthetic data causes collapse, and AI pollution is degrading the training commons. But there are domains where synthetic data genuinely helps. Understanding where and why illuminates the fundamental constraint.

Image augmentation: In computer vision, synthetic data is standard practice. An image can be rotated, flipped, cropped, color-shifted, and noise-added to produce augmented examples. These transformations preserve the label (a rotated cat is still a cat) while increasing apparent dataset size. This works because the transformations are known, controlled, and do not introduce new information—they reveal invariances already present in the data. This is data augmentation, not data creation.

Physics simulations: In robotics, synthetic environments (simulators) generate unlimited training data for robot control policies. A robot arm learning to grasp objects can train in simulation, where thousands of parallel attempts cost nothing. This works because physics engines can accurately model rigid body dynamics, collisions, and sensor noise. The synthetic data is grounded in the same physical laws the robot will encounter in reality. Transfer from simulation to reality (“sim-to-real”) requires domain adaptation but is often successful.

Formal domains: Synthetic data works well in mathematics, logic, and other formal systems. A theorem prover can generate unlimited problem-solution pairs by constructing proofs. A compiler can generate unlimited code-output pairs by executing programs. These synthetic examples are guaranteed correct by construction, because the domain has formal semantics. Models trained on synthetic formal data (e.g., AlphaGeometry, which generates synthetic geometry proofs) achieve strong performance.

Constraint-based generation: When the generation process is constrained by known rules, synthetic data can be valuable. For example, generating SQL queries from schemas, generating chemical formulas that obey valence rules, generating chess positions that obey game rules. The synthetic data is valid by construction, and the model learns the rule structure.

What these successes have in common: 1. The generation process is grounded in known, stable rules (physics, mathematics, formal semantics). 2. The synthetic data is used for augmentation or exploration within a bounded domain, not frontier extension. 3. Correctness can be verified independently (simulation matches reality, proofs are valid, code compiles).

Where synthetic data fails: 1. Open-ended natural language generation: there are no formal rules, no ground truth, no verifier. 2. Frontier knowledge creation: synthetic data cannot contain information not in the generating model. 3. Unverifiable domains: tasks where correctness cannot be checked automatically.

The constraint is clear: synthetic data works when grounded in verifiable structure but fails when asked to extend beyond the learned distribution. For language modeling, where the goal is to match the unbounded diversity of human-generated text, synthetic data leads to collapse unless carefully mixed with fresh human data—which brings us back to the data wall.

The ouroboros is not a solution to the data wall. It is the mechanism by which the wall becomes permanent.

Chapter summary

  • Shumailov et al. (2024) demonstrated that models trained on AI-generated data undergo model collapse: distribution tails vanish, outputs become generic, diversity degrades
  • Collapse happens because models are imperfect estimators; iterating any lossy compression pushes distributions toward lower entropy
  • By generation 4-6 of recursive training, perplexity degrades 60-80%; the collapse accelerates rather than gradual
  • Mixing real and synthetic data prevents regression but does not extend the frontier; synthetic data contains no information not already in the model
  • The information conservation law: you cannot create knowledge by training on your own outputs
  • Data decay is observable now, not future speculation: Stack Overflow (10-30% AI content), Wikipedia (AI edits slip through), arXiv (AI-generated papers), news sites (AI content farms)
  • Estimates suggest 50-90% of new internet text may be AI-generated by 2025-2027; future models will train on polluted data
  • AI detection tools have 5-20% false positive rates; aggressive filtering discards genuine human content along with synthetic noise
  • Synthetic data works in constrained domains: image augmentation, physics simulations, formal systems (math, code), constraint-based generation
  • These successes rely on grounded rules, verifiable correctness, and bounded domains; they do not generalize to open-ended natural language
  • The ouroboros is not a solution; it is the mechanism by which the data wall becomes permanent