🤖 AI

When and How Human Curation Backfires: Preference Alignment under Multi-Model Self-Consuming Loop

This paper reveals that while human curation improves alignment in isolated self-consuming models, extending this paradigm to multi-model systems can cause curation effects to dampen or invert through cross-model interactions, ultimately degrading long-term alignment.

Original authors: Yang Zhang, Xiukun Wei, Xueru Zhang

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yang Zhang, Xiukun Wei, Xueru Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a bustling city where two different types of factories, Factory A and Factory B, are constantly trying to improve their products.

In the old days, these factories would only learn from Real Data: raw materials mined from the earth (like human-written articles or real photos). But recently, a new, cheaper method became popular: Synthetic Data. Instead of mining new materials, the factories started using their own finished products as raw materials to build the next generation of products.

This creates a "self-consuming loop." Factory A makes a product, uses it to train itself, makes a better product, and repeats. The same goes for Factory B.

The Problem: The "Echo Chamber" Effect

The paper explains that if a factory only uses its own recycled products to learn, things start to go wrong. The products get worse, blurrier, or more biased over time. It's like a photocopier copying a copy of a copy; eventually, the image becomes unrecognizable. This is called Model Collapse.

The Proposed Fix: The "Human Editor"

To stop this collapse, researchers suggested adding a Human Editor into the loop. Before the factory uses its own recycled product as training data, a human looks at it and picks only the "good" ones.

In a single factory: This works great! The human editor steers the factory toward making things people actually like.
The Paper's Discovery: The authors asked, "What happens if Factory A and Factory B are talking to each other?"

In the real world, factories don't work in isolation. Factory A might use products made by Factory B to train itself, and vice versa. This is the Multi-Model Ecosystem.

The Big Surprise: When the Editor Backfires

The paper's main finding is shocking: In a connected system, adding more Human Editors doesn't always help. Sometimes, it makes things worse.

Here is the analogy of how this happens:

The Conflicting Preferences: Imagine Factory A loves Red products, while Factory B loves Blue products.
The Cross-Pollination: Factory A starts using Factory B's Blue products to train itself.
The Editor's Mistake: A Human Editor for Factory A looks at the mix of Red and Blue products. They pick the "best" ones based on their own taste.
The Backfire: Because Factory A is learning from Factory B's Blue products, the "best" Blue products might actually push Factory A's internal logic in a direction that hurts its ability to make Red products.
- The Editor thinks they are helping by picking the "best" data.
- But because of the complex connection between the two factories, that "best" data actually confuses Factory A.
- Result: The more the Editor tries to curate the data, the more Factory A drifts away from making what its users actually want.

The "Sensitivity" Metaphor

The paper uses a concept called Sensitivity Matrices to explain this. Think of the two factories as two people standing on a trampoline.

If you push Factory A (by curating its data), it bounces up.
But because they are on the same trampoline, Factory A's bounce pushes Factory B down.
Factory B then bounces back up, hitting Factory A from a different angle.
The paper shows that these "bounces" can amplify, cancel out, or even reverse the original push. You might push Factory A to go North, but the trampoline dynamics push it South.

Key Takeaways from the Paper

Isolation vs. Connection: In a single, isolated factory, human curation always improves the product. In a connected ecosystem of interacting factories, human curation can have non-monotonic effects (meaning more curation doesn't equal better results; it can get worse).
The "Preference Domain Mismatch": Sometimes, the data Factory A learns from (Blue products) is completely different from what its customers actually want to see (Red products). Even if the Human Editor picks the "best" Blue products, it doesn't help Factory A make better Red products. The effort is wasted or harmful because the training data doesn't match the real-world goal.
Stability: The paper proves that if you keep enough Real Data (fresh raw materials) in the mix, the system stays stable and doesn't collapse. But if you rely too much on recycled data and human curation in a complex web of interactions, the system can become unstable and degrade.

Summary

The paper warns us that as AI models start training on data generated by other AI models, we can't just assume that "human review" will fix everything. In a complex network of interacting models, human curation can accidentally steer the system in the wrong direction, creating a situation where trying harder to align the AI with human preferences actually makes it less aligned.

Technical Summary: When and How Human Curation Backfires

Problem Statement

Foundation models are increasingly trained on synthetic data generated by prior model iterations, creating "self-consuming" loops. While recent work suggests that incorporating human curation into these loops can steer models toward human-aligned behavior, existing analyses largely focus on isolated, single-model settings. In practice, data ecosystems are inherently multi-model: models interact, and their outputs are recycled to train other models (e.g., text-to-image and image-to-text models feeding into one another, or instruction-tuning pipelines where one model's outputs become another's training data).

The paper addresses a critical gap in understanding: How does human curation affect preference alignment in a multi-model self-consuming ecosystem? Specifically, it investigates whether increasing the fraction of human-curated synthetic data for one model consistently improves alignment for that model and the broader system, or if cross-model interactions can distort, dampen, or even invert these effects.

Methodology

The authors formalize a theoretical framework for interacting self-consuming models and analyze the system's dynamics using tools from optimization theory and performative prediction.

1. Framework Formalization

The study considers a two-model ecosystem (models $\theta$ and $\phi$ ) where each model is iteratively updated on a mixture of:

Real data ( $R$ ): Fixed distributions.
Self-consuming synthetic data ( $S$ ): Generated by the model itself or the other model.
Human-curated synthetic data ( $H$ ): Synthetic samples refined by human preference (modeled via a generalized Bradley-Terry model or Direct Preference Optimization).

The training data distribution for each model at round $t+1$ is a weighted mixture of these sources. The models update their parameters to minimize expected loss functions ( $\ell_\theta, \ell_\phi$ ) under these distributions. The framework accounts for both synchronous and asynchronous update regimes.

2. Stability and Convergence Analysis

The authors first establish conditions under which the multi-model system converges to a stable point $(\theta^*, \phi^*)$ .

Assumptions: The loss functions are assumed to be strongly convex and smooth, and the induced data distributions are assumed to be Lipschitz continuous with respect to model parameters (distribution sensitivity).
Result: They prove that if the fraction of real data in each training round is sufficiently large (exceeding a threshold $\tau$ ), the iterative training loop converges to a unique stable point. This extends prior single-model convergence results to the multi-model regime, showing that cross-model interactions do not disrupt convergence provided real data anchors the system.

3. Quantifying Curation Impact (Local Sensitivity)

Once convergence is established, the paper analyzes the long-term impact of human curation on preference alignment. They define alignment metrics $J_p(\theta)$ and $J_q(\phi)$ as the expected reward under evaluation distributions.

Decomposition: The impact of increasing the curation fraction $\lambda_H$ $λ_{H}$ for model $\theta$ $θ$ is decomposed into:
- Self-influence: The direct effect on model $\theta$ 's alignment.
- Cross-influence: The propagated effect on model $\phi$ 's alignment.
Mathematical Formulation: Using the implicit function theorem and chain rule, the authors derive expressions for the derivatives $\frac{\partial J_p}{\partial \lambda_H}$ $\frac{\partial J _{p}}{\partial λ _{H}}$ and $\frac{\partial J_q}{\partial \lambda_H}$ $\frac{\partial J _{q}}{\partial λ _{H}}$ . These expressions involve:
- Sensitivity Matrices ( $S_p, S_q$ ): Characterizing how a model's stable point responds to perturbations, accounting for coupled dependencies.
- Cross-model Influence Matrices ( $C_p, C_q$ ): Quantifying how parameter shifts in one model propagate to the other.
- Alignment Direction ( $\rho$ ): The cosine similarity between the curation-induced update direction and the reward-improving direction.

4. Theoretical Findings on "Backfiring"

The core theoretical contribution is the demonstration that increasing human curation does not necessarily improve alignment in multi-model settings.

Non-Monotonicity: Unlike in isolated settings where curation consistently improves alignment (assuming $\rho > 0$ ), in multi-model systems, the sensitivity matrices ( $S$ ) and cross-influence matrices ( $C$ ) can distort the update direction.
Inversion Mechanism: Even if the curation direction is locally aligned with the reward ( $\rho > 0$ ), the coupled dynamics can invert the effect, leading to $\frac{\partial J}{\partial \lambda_H} < 0$ . This occurs when cross-model interactions amplify discrepancies or when the "preference domain mismatch" (PDM) causes training on one domain to degrade performance on the evaluation domain.
Global Deviation: The paper also quantifies the global alignment gap, showing that the deviation from an ideal real-data-only training scenario scales linearly with the amount of synthetic data, provided the system remains stable.

Key Contributions

Unified Framework: Formalized an analytical framework for interacting self-consuming models that incorporates human preference feedback without relying on stylized distributional assumptions.
Convergence Conditions: Provided rigorous conditions for the stability and convergence of multi-model self-consuming systems, highlighting the role of real-data ratios in maintaining stability against cross-model interactions.
Decomposition of Curation Effects: Quantified the long-term impact of human curation by decomposing it into self-influence and cross-influence. This allows for the identification of conditions where curation improves or degrades alignment.
Identification of Backfire Conditions: Demonstrated theoretically and empirically that cross-model interactions can dampen or reverse the benefits of human curation, leading to degraded long-term alignment even when curation is intended to improve it.

Experimental Validation

The authors validated their theoretical findings through three experimental settings:

Gaussian Models: A controlled synthetic environment extending Example 4.6 to validate the mechanism of sensitivity matrix distortion. Results confirmed that $\rho > 0$ does not guarantee positive curation effects and that the theoretical derivatives match empirical observations.
CIFAR-10 (Vision): Two class-conditional diffusion models with conflicting preferences (warm-toned vs. cool-toned images). Experiments showed that increasing curation for one model could degrade its own alignment and that of the other model when cross-model data fractions were high, confirming the non-monotonic behavior.
Qwen2.5-0.5B (Language): Models trained on summarization vs. paraphrasing tasks. This experiment highlighted Preference Domain Mismatch (PDM), where cross-model training inputs mismatched the evaluation domains, effectively masking the coupling effects on the target reward metrics.

Significance and Claims

The paper claims to provide a mechanistic and analyzable understanding of how multi-model self-consuming ecosystems evolve and how human curation influences them.

Theoretical Insight: It challenges the intuition that "more human curation is always better" in complex, interconnected data ecosystems. It reveals that model interactions can create feedback loops where curation intended for one model inadvertently harms the alignment of the system.
Practical Implication: The findings suggest that in multi-model pipelines (e.g., instruction tuning, multimodal data generation), simply increasing curation ratios without accounting for cross-model dependencies and preference conflicts may lead to suboptimal or even degraded alignment.
Scope: The authors position this work as a foundational step toward understanding the stability and alignment of coupled AI systems, offering local diagnostics (via sensitivity matrices) to assess whether curation interventions are likely to be neutralized or reversed by ecosystem externalities.

The paper remains modest in its claims, acknowledging limitations such as the reliance on standard regularity assumptions (strong convexity, smoothness) which may not fully capture the non-convex landscape of modern large-scale neural networks, and the difficulty of estimating high-dimensional sensitivity matrices in practice. It frames the work as a theoretical exploration of "stress tests" and existence proofs for non-monotonic curation effects.