Technical Summary: Convergence and Stability Analysis of Self-Consuming Generative Models with Heterogeneous Human Curation

Problem Statement

The paper addresses the theoretical stability and convergence of self-consuming generative models, where a model is iteratively retrained on a mixture of real data and its own previously generated synthetic outputs. While such "curate-and-retrain" loops are central to modern alignment pipelines (e.g., RLHF, DPO), existing theoretical frameworks often rely on idealized assumptions: homogeneous, noise-free human preferences and deterministic selection of the "best" candidate.

The authors identify two critical gaps in prior work (specifically Ferbach et al., 2024):

1. Heterogeneity: Real-world curation involves raters with diverse preferences and stochastic noise, which prior models often ignore.
2. Regularization: Practical systems frequently mix curated synthetic data with a fixed reference distribution (e.g., the initial pre-trained model or real data) to prevent collapse. However, the convergence and robustness properties of these mixed-data regimes had not been rigorously established, particularly under heterogeneous preferences.

The paper investigates the asymptotic behavior of retraining dynamics across four regimes defined by the size of the candidate pool ($K$) and the mixing weight of the reference distribution ($\alpha$).

Methodology

Model Setup

The authors formalize a discrete-time dynamical system where, in each round $t$:

1. Generation: The model $p_t$ generates $K$ i.i.d. candidates $X_{1:K}$.
2. Curation: A heterogeneous human curator selects one candidate $\hat{X}$ based on a random-utility model. The utility of candidate $x_k$ is $\tilde{w}_k = \exp(r(x_k) + \varepsilon_k(x_k))$, where $r$ is a reward function and $\varepsilon_k$ represents heterogeneous preference noise (capturing both inter-rater differences and intra-rater stochasticity). Selection follows the Plackett–Luce rule:
   $$P(I=k | x_{1:K}, \varepsilon_{1:K}) = \frac{\tilde{w}_k}{\sum_{j=1}^K \tilde{w}_j}$$
3. Update: The new distribution $p_{t+1}$ is derived by maximizing a log-likelihood objective that mixes the curated sample distribution with a fixed reference distribution $p_{\text{ref}}$ with weight $\alpha \in [0, 1)$.

Four Regimes

The analysis covers four distinct settings:

• Regime (i): $\alpha = 0$ (pure synthetic), $K < \infty$ (finite pool).
• Regime (ii): $\alpha = 0$ (pure synthetic), $K = \infty$ (infinite pool/mean-field limit).
• Regime (iii): $\alpha \in (0, 1)$ (mixed), $K < \infty$.
• Regime (iv): $\alpha \in (0, 1)$ (mixed), $K = \infty$.

Analytical Tools

• Closed-Form Dynamics: The authors derive exact population-level update rules for all four regimes, characterizing the "choice kernel" that maps the current distribution to the next.
• Contraction Mapping (Regime iii): For finite pools with mixing, the authors utilize the Banach Fixed Point Theorem in the Total Variation (TV) metric. They establish conditions under which the update operator is a contraction.
• Nonlinear Perron–Frobenius Theory (Regime iv): For infinite pools with mixing, the TV metric fails to show contraction. Instead, the authors employ the Hilbert projective metric and nonlinear Perron–Frobenius theory to prove convergence to a unique fixed point.
• Stability Analysis: The paper evaluates sensitivity to bounded reward perturbations ($\Delta r$) to determine if the retraining trajectory remains close to the unperturbed path.

Key Contributions

1. Heterogeneous Rater Model: The paper generalizes the self-consuming framework to include between-rater heterogeneity and within-rater noise via a random-utility model, moving beyond the deterministic, homogeneous curator assumption of prior work.
2. Closed-Form Dynamics: Exact population update formulas are derived for both finite and infinite candidate pools, with and without reference mixing.
3. Convergence Guarantees:
   • Pure Synthetic ($\alpha=0$): Proves convergence to the distribution supported on the maximum reward set in terms of Kullback–Leibler (KL) divergence.
   • Mixed Finite Pool ($\alpha > 0, K < \infty$): Establishes geometric convergence in TV distance provided the mixing weight $\alpha$ exceeds a threshold ($\alpha > \frac{K-1}{K}$).
   • Mixed Infinite Pool ($\alpha > 0, K = \infty$): Proves convergence in the Hilbert projective metric to an explicit fixed point, a stronger result than KL or TV convergence.
4. Stability and Robustness:
   • Demonstrates that pure synthetic retraining ($\alpha=0$) is unstable under arbitrarily small bounded reward perturbations; the limiting distribution can shift entirely to a different support set.
   • Proves that regularized retraining ($\alpha > 0$) is Lipschitz-stable (or stable in the limit) under bounded perturbations, provided the mixing weight is sufficient.

Results

Convergence

• Regimes (i) & (ii): The sequence of distributions $\{p_t\}$ converges to $p^*$, which is the initial distribution $p_0$ restricted to the set of points maximizing the expected exponential reward $Q(x)$. Convergence is monotonic in expected reward and occurs in KL divergence.
• Regime (iii): If $\alpha > \frac{K-1}{K}$, the update operator is a contraction in TV distance with rate $\rho = K(1-\alpha)$. This guarantees a unique fixed point and geometric convergence.
• Regime (iv): Under Assumption A3 (a condition on $\alpha$ relative to the reward landscape), the dynamics converge in the Hilbert projective metric to a unique fixed point $p^*$. This metric is stronger than TV and KL, implying convergence in those metrics as well.

Stability

• Instability of Pure Loops: In Regimes (i) and (ii), the authors construct a counterexample showing that a small perturbation in the reward function can cause the limiting distribution to shift from set $A$ to a disjoint set $A_{\Delta r}$, resulting in a TV distance of 1.
• Stability of Regularized Loops: In Regimes (iii) and (iv), the presence of the reference distribution $p_{\text{ref}}$ acts as a stabilizer. The authors prove that the distance between the perturbed and unperturbed trajectories is bounded by a term proportional to the perturbation magnitude $\|\Delta r\|_\infty$. This establishes superalignment (improvement in reward + trajectory stability).

Significance and Claims

The paper claims to provide the first rigorous convergence and stability analysis for self-consuming generative models under heterogeneous human preferences and reference mixing.

• Theoretical Advancement: It extends the work of Ferbach et al. (2024) by relaxing the assumption of homogeneous, noise-free curators and by establishing convergence for mixed-data settings, which were previously an open problem.
• Practical Insight: The analysis highlights that reference mixing is not merely a heuristic but a theoretical necessity for stability. Without mixing ($\alpha=0$), the system is fragile to reward misspecification. With sufficient mixing, the system achieves robustness and geometric convergence.
• Methodological Novelty: The application of nonlinear Perron–Frobenius theory and the Hilbert projective metric to generative model retraining dynamics offers a new analytical toolkit for studying non-contractive, non-linear population updates.

The authors maintain a modest tone regarding future work, noting that characterizing the explicit form of the fixed point in Regime (iii) remains open, and the sharpness of convergence rates at critical thresholds requires further investigation. They also note that their results extend verbatim to conditional generative models.