Preventing Model Collapse Under Overparametrization: Optimal Mixing Ratios for Interpolation Learning and Ridge Regression

1. Problem Statement

The paper addresses model collapse, a phenomenon where generative models degrade in performance when iteratively trained on their own synthetic outputs. While previous literature (e.g., Shumailov et al., 2024) established that model collapse occurs in low-dimensional settings or when real data is fixed and not refreshed, there was a significant gap in understanding this dynamic in overparameterized regimes (where the number of features $p$ exceeds the number of samples $n$).

The authors investigate whether mixing fresh real data with synthetic data at every iteration can prevent collapse in high-dimensional linear regression. Specifically, they ask:

1. Can mixing fresh real labels with synthetic labels mitigate model collapse in overparameterized linear regression?
2. What is the optimal mixing ratio ($w$) of real-to-synthetic data that minimizes long-term prediction error?

2. Methodology and Framework

The authors analyze two primary classes of estimators under an iterative "fresh data augmentation" framework:

• Minimum-$\ell_2$-norm Interpolators: The limit of gradient descent on overparameterized models with no explicit regularization.
• Ridge Regression: Explicitly regularized estimators.

The Iterative Setup:
At iteration $t$, the model is trained on a weighted mixture of:

• Fresh Real Responses ($y_t$): Generated from the true model $y_t = X\beta + \epsilon_t$.
• Synthetic Responses ($\tilde{y}_t$): Generated using the estimator from the previous iteration $\hat{\beta}_{t-1}$, i.e., $\tilde{y}_t = X\hat{\beta}_{t-1} + \tilde{\epsilon}_t$.

The estimator update rule is:
$$\hat{\beta}_t = (X^\top X)^\dagger X^\top (w y_t + (1-w)\tilde{y}_t)$$
$$\hat{\beta}_{t,\lambda} = (X^\top X + n\lambda I)^{-1} X^\top (w y_t + (1-w)\tilde{y}_{t,\lambda})$$
where $w \in (0,1)$ is the mixing weight for real data.

Assumptions:

• High-Dimensional Regime: $p, n \to \infty$ with $p/n \to \gamma > 1$.
• Covariates: $X = Z\Sigma^{1/2}$, where $Z$ has i.i.d. entries and $\Sigma$ has bounded eigenvalues.
• Signal: The signal strength $\|\beta\|^2$ converges to a constant $b^*$.
• Noise: Independent Gaussian-like noise with bounded moments.

3. Key Contributions and Theoretical Results

A. Generalization Error Characterization

The authors derive precise asymptotic formulas for the generalization error (risk) as $t \to \infty$ and $n \to \infty$. The risk decomposes into Bias and Variance components, both of which depend on the mixing weight $w$, the regularization $\lambda$, and the spectral properties of the covariance matrix $\Sigma$.

B. Optimal Mixing for Min-$\ell_2$-Norm Interpolation

For the unregularized interpolator ($\lambda = 0$):

• Golden Ratio Convergence: The authors prove that the optimal mixing weight $w^*$ that minimizes the asymptotic risk converges to the reciprocal of the golden ratio:
  $$w^* = \frac{1}{\phi} = \frac{\sqrt{5}-1}{2} \approx 0.618$$
• Robustness: This result holds for any feature covariance matrix $\Sigma$ with bounded eigenvalues, extending previous low-dimensional findings to the overparameterized regime.
• Collapse Prevention: If $w = 0$ (training only on synthetic data), the risk diverges to infinity (model collapse). If $w > 1/3$, the variance decreases monotonically with iterations, preventing collapse. The optimal $w^* \approx 0.618$ ensures the fastest convergence to the minimal risk.

C. Optimal Mixing for Ridge Regression

For ridge regression ($\lambda > 0$), the analysis is more complex as both bias and variance depend on $w$.

• Log-Convexity: The limiting risk is proven to be a log-convex function of $w$, guaranteeing a unique global minimum.
• Lower Bound: Across various models (isotropic, spiked covariance, random effects), the optimal weight satisfies $w^* \geq 0.5$. This implies that real data must always be weighted more heavily than synthetic data to prevent collapse.
• Dependence on Parameters:
  • As $\lambda \to 0$ (approaching interpolation), $w^* \to 1/\phi \approx 0.618$.
  • As $\lambda \to \infty$ (strong regularization), $w^* \to 1$ (relying entirely on real data).
  • $w^*$ increases with the Signal-to-Noise Ratio (SNR).

D. Specific Model Classes

The paper analyzes two specific structural assumptions to provide concrete insights:

1. Random Effects Model: Where signal components are i.i.d. The optimal mixing ratio follows the general bounds ($w^* \in [0.5, 1]$).
2. Spiked Covariance Model: Where the covariance matrix has a few large eigenvalues. The results confirm that spectral geometry governs the optimal weighting, but the necessity of $w^* \geq 0.5$ remains.

E. Alternative Settings

The authors explore scenarios where the "fresh data" assumption is relaxed:

• No Fresh Real Data: If real labels are fixed and only synthetic data is added iteratively, model collapse is inevitable unless $w=1$ (no synthetic data used).
• Time-Varying Covariates: If $X$ changes every iteration but fresh labels are available, the optimal $w^*$ depends on $\gamma$ and is generally non-monotonic.
• Partial Fresh Labels: If only a fraction of covariates receive fresh labels, model collapse can be inevitable in highly overparameterized regimes ($\gamma > 2$) regardless of mixing.

4. Significance and Implications

• Theoretical Breakthrough: This is one of the first rigorous analyses of model collapse in overparameterized settings. It bridges the gap between low-dimensional collapse theories and modern high-dimensional deep learning practices.
• Actionable Guidance: The paper provides a concrete, mathematically derived rule for practitioners: Mix approximately 62% real data with 38% synthetic data (for interpolation tasks) to minimize long-term error. For regularized models, real data should constitute at least 50% of the training mix.
• Mechanism of Collapse: The analysis reveals that collapse is driven by the accumulation of variance when synthetic data dominates. The "Golden Ratio" weighting acts as a critical threshold that balances the bias-variance trade-off in the recursive learning loop.
• Limitations of Synthetic Data: The results suggest that while synthetic data can augment learning, it cannot replace real data in iterative training loops without causing degradation, especially in high-dimensional regimes.

5. Validation

The theoretical findings are validated through extensive simulations. The empirical risk curves closely match the derived asymptotic formulas for various covariance structures (isotropic, AR(1), spiked) and sample sizes, confirming the robustness of the $w^* \approx 0.618$ and $w^* \geq 0.5$ findings.

Conclusion

The paper establishes that model collapse in overparameterized linear regression is preventable through a specific mixing strategy. By weighting fresh real data at approximately 61.8% (the reciprocal of the golden ratio) for interpolation, or at least 50% for regularized models, one can ensure the generalization error remains bounded and minimized, effectively breaking the "curse of recursion."