Curated Synthetic Data Doesn't Have to Collapse: A Theoretical Study of Generative Retraining with Pluralistic Preferences

Technical Summary: Curated Synthetic Data Doesn't Have to Collapse

Problem Statement
The paper addresses the critical challenge of mode collapse in recursive retraining of generative models. As reliance on synthetic data increases to overcome bottlenecks in human-generated data, models are often retrained on curated subsets of their own outputs. Prior work (e.g., Ferbach et al., 2024) demonstrated that when curation is driven by a single fixed reward function, the model distribution inevitably collapses onto a narrow set of outputs that over-optimize that specific objective. This phenomenon occurs because the selection process (modeled as a Bradley-Terry choice) exponentially reweights the distribution toward the global maximizer of the single reward, eventually eliminating support for other regions. The paper questions whether this collapse is unavoidable when curation is based on pluralistic (multiple, potentially conflicting) preferences, a scenario common in real-world applications where stakeholders value different traits (e.g., factual accuracy vs. creativity).

Methodology
The authors formalize recursive retraining under heterogeneous preferences using a theoretical framework that combines:

1. Multi-Preference Selection: At each retraining step, a reward function is chosen probabilistically from a fixed mixture (e.g., $r_1$ with probability $q$, $r_2$ with probability $1-q$).
2. Bradley-Terry (BT) Curation: Given a set of $K$ candidates sampled from the current model distribution $p_t$, a single sample is selected based on the chosen reward $r$ using the BT model, which assigns selection probabilities proportional to $e^{r(x)}$.
3. Idealized MLE Update: The model is retrained via exact Maximum Likelihood Estimation (MLE) on the selected sample, effectively setting the next distribution $p_{t+1}$ to the distribution of the selected sample.
4. Large-Pool Limit Analysis: The authors analyze the dynamics in the limit of large candidate pools ($K \to \infty$), where the selection weight converges to an exponential tilt of the current model. They introduce $\epsilon$-optimal basins ($S_{i,\epsilon}$) representing regions where rewards are near their maximum and define leakage factors ($\kappa$) to quantify cross-reward reinforcement (how much one reward favors the other's basin).

Key Contributions
The paper makes three primary theoretical contributions:

1. Pluralistic Retraining Dynamics: The authors formalize that under heterogeneous preferences, the update rule acts as a convex combination of exponential tilts. They prove that if an "outside-basin gap" exists (regions outside the optimal basins are uniformly worse), probability mass contracts into the union of the near-optimal basins ($S_\epsilon$), preventing total collapse to a single point.
2. Leakage-Controlled Limit Structure: The study characterizes the limiting distribution as a mixture supported on the union of the competing basins. The weight of this mixture is constrained by leakage factors. In the hard-separation regime (where basins are disjoint and cross-reward leakage vanishes), the limiting mixture weight converges exactly to the reward sampling probability $q$.
3. Non-Collapse and Value Aggregation: The paper proves that the limiting distribution preserves diversity, evidenced by strictly positive reward variance. Furthermore, it establishes that the limiting mixture satisfies a weighted Nash bargaining solution, providing a formal interpretation of how conflicting objectives are aggregated in synthetic retraining loops.

Results
Theoretical and empirical results confirm the following:

• Diversity Preservation: Unlike single-reward retraining, pluralistic curation prevents the distribution from collapsing to a single mode. The limiting distribution maintains non-trivial mass on multiple high-reward regions.
• Phase Transitions: Empirical experiments on synthetic Gaussian mixtures show a phase transition based on the distance between reward optima. When optima are well-separated, the model stabilizes as a two-mode mixture with positive variance. As the distance decreases, the variance drops, and the model collapses toward a compromise region.
• CIFAR-10 and Text Experiments:
  • On CIFAR-10, increasing the number of balanced preferences (from 2 to 5) improves sample quality (lower FID) and diversity (higher entropy, lower KL divergence). Conversely, imbalanced preferences (high polarization) lead to degradation and partial collapse.
  • In text generation (GPT-2) with conflicting length preferences, increasing the distance between target lengths leads to higher sustained entropy in the output distribution, indicating the model maintains a policy hedging between objectives rather than collapsing to a single compromise length.

Significance and Claims
The paper claims to overturn the prevailing conclusion that synthetic retraining inevitably leads to collapse. It argues that collapse is not inherent to the use of synthetic data, but rather a consequence of how that data is curated. By employing pluralistic preferences, the retraining loop acts as a regularizer that resists overfitting to narrow alignment signals.

The authors position their work as a theoretical foundation for understanding multi-objective alignment in practices like Reinforcement Learning from Human Feedback (RLHF). They suggest that pluralistic curation offers a theoretically sound mechanism for aligning generative models with multifaceted human values without sacrificing diversity or quality, particularly in scenarios where human oversight data is scarce. The work provides a formal justification for using mixed-reward strategies to maintain robustness and representational breadth in generative models.