Epidemiology of Model Collapse: Modeling Synthetic Data Contamination via Bilayer SIR Dynamics

Technical Summary: Epidemiology of Model Collapse: Modeling Synthetic Data Contamination via Bilayer SIR Dynamics

Problem Statement
The paper addresses the phenomenon of "model collapse," where training large language models (LLMs) on synthetic data leads to progressive degradation in output quality and diversity. While prior work (e.g., Shumailov et al. [20], Dohmatob et al. [4]) has characterized this as a single-chain degradation process (Model A $\rightarrow$ Data $\rightarrow$ Model B $\rightarrow$ Data), the authors argue that the real AI ecosystem is a network characterized by cross-contination. In this ecosystem, thousands of models ingest data from shared corpora, and their outputs re-enter the data pool via web publishing and APIs. The paper posits that existing mathematical frameworks fail to capture these ecosystem-level dynamics, specifically the feedback loop where contaminated data "infects" models, which in turn "transmit" synthetic artifacts back into data corpora.

Methodology
The authors propose a phenomenological mean-field model treating the AI ecosystem as a bilayer coupled epidemic system, analogous to infectious disease spreading.

1. Bilayer SIR/SIRS Framework:

   • The system consists of two interacting populations: Data Corpora (Layer D) and AI Models (Layer M).
   • Each layer is divided into Susceptible (S), Infected (I), and Recovered (R) compartments.
   • Data Layer: Clean data ($S_D$) becomes contaminated ($I_D$) when synthetic content is ingested; it recovers ($R_D$) via detection and filtering.
   • Model Layer: Clean-trained models ($S_M$) become contaminated ($I_M$) when trained on contaminated data; they recover ($R_M$) via retraining on verified-clean data.
   • Cross-Layer Transmission: Contaminated models ($I_M$) infect data at rate $\beta_D$, and contaminated data ($I_D$) infects models at rate $\beta_M$.
   • SIRS Variant: The authors recommend the SIRS variant over SIR, incorporating immunity waning ($\delta$), reflecting that filtered corpora and retrained models remain susceptible to re-contamination.

2. Mathematical Analysis:

   • The dynamics are governed by six coupled ordinary differential equations (ODEs).
   • The Basic Reproduction Number ($R_0$) is derived using the Next Generation Matrix method. The authors prove that $R_0 = \sqrt{\frac{\beta_D \beta_M}{(\gamma_D + \mu_D)(\gamma_M + \mu_M)}}$, where $\gamma$ represents recovery/removal rates and $\mu$ represents turnover/obsolescence rates.
   • The paper establishes the existence and stability of the Disease-Free Equilibrium (DFE) and Endemic Equilibrium (EE), demonstrating a transcritical bifurcation at $R_0 = 1$.

3. Calibration and Sensitivity:

   • Parameters are estimated from public data on AI text prevalence (2023–2025) and detection accuracy.
   • Three scenarios (Optimistic, Baseline, Pessimistic) are defined.
   • Global sensitivity analysis is performed using Sobol indices to identify high-leverage parameters.

4. Empirical Validation:

   • Agent-Based Model (ABM): A stochastic bipartite network simulation is used to check consistency with the mean-field ODE.
   • GPT-2 Experiments:
     • Single-chain: 192 runs measuring dose-response degradation in perplexity (PPL) and diversity (Distinct-2).
     • Source-Diversity: 1,088 matched-budget runs testing whether mixing synthetic data from multiple sources ($K > 1$) attenuates collapse compared to self-training ($K=1$).

Key Contributions

1. Bilayer SIR/SIRS Framework: A phenomenological ODE model coupling data and model populations, with SIRS recommended to account for re-contamination.
2. Threshold Analysis: Derivation of $R_0$ via the Next Generation Matrix, along with proofs for DFE stability, endemic equilibrium existence, and transcritical bifurcation.
3. Illustrative Calibration: Scenario-based calibration yielding supercritical dynamics ($R_0 > 1$) across plausible parameter ranges, with Sobol analysis identifying synthetic-text detection ($\gamma_D$) as the highest-leverage parameter.
4. GPT-2 Experiments: Demonstration of dose-response degradation and diversity loss in single-chain runs; investigation of source diversity effects in matched-budget experiments.
5. Intervention Analysis: Evaluation of six strategies (e.g., watermarking, herd immunity) showing that detection-based filtering is the most effective lever for driving $R_0 < 1$.

Results

• Supercritical Dynamics: All three calibration scenarios yield $R_0 > 1$ (Baseline $R_0 = 2.62$). Uncertainty propagation indicates a 98.2% probability of supercritical dynamics under assumed parameter ranges.
• Sensitivity: Sobol sensitivity analysis identifies data recovery/detection rate ($\gamma_D$) as the most influential parameter ($S_T = 0.324$), followed by data contamination rate ($\beta_D$) and model recovery rate ($\gamma_M$). Turnover rates ($\mu_D, \mu_M$) have negligible impact.
• ODE-ABM Consistency: The mean-field ODE agrees strongly with the ABM ensemble mean in dense networks ($R^2 > 0.96$) but degrades under network heterogeneity (e.g., superspreaders).
• GPT-2 Single-Chain: Recursive training on self-generated data causes significant degradation. Distinct-2 bigram diversity drops from 0.68 to 0.38 under full contamination ($\alpha=1.0$).
• Source-Diversity Experiments:
  • At full contamination ($\alpha=1.0$), multi-source mixing ($K=3, 5$) provides a modest attenuation of collapse (~2 PPL reduction, $d \approx 0.8$, one-sided $p=0.047$) compared to self-training ($K=1$).
  • At partial contamination ($\alpha=0.5$), the diversity buffer vanishes ($p=0.61$). The presence of 50% real data breaks the self-reinforcing loop, making source diversity irrelevant.
• Intervention Analysis: Under baseline parameters, only watermark-based filtering and herd immunity achieve $R_0 < 1$ individually. Detection combinations dominate in pairwise evaluations.

Significance and Claims
The paper claims to provide a principled epidemiological vocabulary ($R_0$, herd immunity, intervention thresholds) for reasoning about AI ecosystem contamination. It argues that the cross-contamination process is structurally analogous to epidemic spreading and that the bilayer framework captures the critical cross-layer leverage missing from single-chain models.

The authors emphasize that their framework is phenomenological and illustrative, not a precise measurement of the real ecosystem. They conclude that the dominant driver of model collapse is the contamination fraction ($\alpha$), and that practical interventions should prioritize detection and filtering (reducing $\alpha$) over diversifying contamination sources. The modest benefit of source diversity observed at $\alpha=1$ vanishes at realistic contamination levels, confirming that $\beta_D$ and $\gamma_D$—not ecosystem structure—are the critical intervention levers. The paper explicitly notes limitations, including the stylized nature of the mean-field ODE, the small scale of GPT-2 experiments, and the conditional nature of the calibration results.