Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gain

1. Problem Statement

Current Large Language Model (LLM) self-evolving systems, often framed as self-play (where a model acts as a Proposer, Solver, and Verifier), frequently suffer from mode collapse or rapid performance plateaus.

• The Core Failure: Existing loops synthesize increasing amounts of data but fail to increase the learnable information available to the model in subsequent iterations.
• The Symptom: The system enters a state of "self-delusion," generating trivial tasks (e.g., $f(x)=x$) or noise that the model cannot learn from, leading to a decline in overall performance.
• The Gap: Traditional approaches rely on reward shaping or fixed datasets, which do not guarantee a monotonic increase in the structural complexity that a bounded observer (the model) can actually learn.

2. Theoretical Framework: Learnable Information

The authors reframe self-evolution not as a game, but as a self-synthetic data pipeline. They introduce an information-theoretic perspective based on Epiplexity (Epistemic Complexity).

• Learnable vs. Unlearnable Information:
  • Learnable Information ($S_{C,T}$): Reusable structure in data that a model with specific capacity ($C$) and inference-time budget ($T$) can compress or predict.
  • Unlearnable Information ($H_{C,T}$): Residual randomness or noise that appears unpredictable given the model's constraints.
• The "Goldilocks Zone": Sustainable evolution requires data that is neither too simple (low structure) nor too hard (high noise) for the current observer. The system must continuously generate data within this zone where structure is complex enough to be non-trivial but learnable.
• Formalism: The paper uses a Bounded Minimum Description Length (MDL) objective to quantify this. The goal is to maximize the reduction in description length (structure extraction) relative to the observer's budget.

3. Methodology: Three System-Level Design Principles

To achieve genuine self-evolution, the paper proposes a triadic framework (Proposer, Solver, Verifier) governed by three necessary design principles:

A. Asymmetric Co-evolution

• Concept: Exploits the computational asymmetry between solving (hard) and proposing/verifying (easier).
• Mechanism:
  • Weak-to-Strong: The Proposer/Verifier (internal environment) generates tasks that are slightly harder than the current Solver's capability, creating a learning gradient.
  • Strong-to-Weak (Crucial Innovation): As the Solver improves, the Proposer and Verifier must be synchronized to track this new frontier. If the internal environment remains static, the tasks become trivial relative to the Solver, causing collapse.
• Implementation: Organizing synthetic directions by "asymmetry gaps" (e.g., moving from simple grammar correction to complex math proofs) and using techniques like back-translation or shared optimization objectives to update the Proposer/Verifier.

B. Capacity Growth

• Concept: The amount of learnable information in self-synthesized data is relative to the observer's capacity. A fixed-capacity model will eventually saturate.
• Mechanism: The system must dynamically expand its Parameter Budget ($C$) and Inference-Time Budget ($T$) as iterations progress.
• Implementation:
  • Parameter Growth: Scaling up the model size or activating more experts/layers over time.
  • Inference Growth: Increasing reasoning depth (e.g., Chain-of-Thought length) or tool usage per iteration.
  • Without this, the model resorts to memorization rather than learning new structures.

C. Proactive Information Seeking

• Concept: A closed loop is bounded by the information already present in the pre-trained weights. To avoid saturation, the system must actively seek external context.
• Mechanism: The internal environment (Proposer/Verifier) must proactively select external data sources (documents, tools, environments) that align with the Solver's current frontier.
• Implementation:
  • Learn to Ask: Generating queries based on Solver failures or Verifier disagreements.
  • Context as Asymmetry: Using retrieved context to create new synthetic directions (e.g., grounding answers in retrieved documents) rather than just using it as a static hint.
  • Co-evolution: Updating retrieval and memory mechanisms alongside the Solver.

4. Experimental Results

The authors conducted diagnostic experiments using code-based tasks (Induction, Abduction, Deduction) to validate their theory using Prequential MDL to estimate Epiplexity.

• Experiment 1 (Capacity & Direction):
  • Finding: Stronger Proposers generate data with higher learnable information.
  • Finding: Learnable information follows an inverted-U curve relative to Solver size. If the Solver is too small, it cannot learn the structure; if too large (relative to the task complexity), it resorts to memorization, reducing the measured learnable information.
  • Finding: Different synthetic directions yield vastly different information gains (Induction > Abduction > Deduction).
• Experiment 2 (Self-Play Iterations):
  • Finding: Without the proposed mechanisms, standard self-play leads to fluctuating and declining learnable information over iterations.
  • Observation: The system collapses into trivial patterns, confirming that reward optimization alone is insufficient for sustained evolution.

5. Key Contributions

1. Redefinition of Self-Evolution: Shifts the paradigm from "Self-Play" (a static RL game) to a "Self-Synthetic Data Pipeline" focused on monotonic learnable information gain.
2. Theoretical Formalization: Introduces Epiplexity and Bounded MDL as metrics to distinguish between learnable structure and unlearnable noise in self-generated data.
3. Triadic Design Principles: Proposes a concrete, system-level architecture requiring Asymmetric Co-evolution, Capacity Growth, and Proactive Information Seeking to prevent collapse.
4. Diagnostic Tool: Provides a method (Prequential MDL) to measure the "health" of a self-evolving loop by quantifying the information gain, rather than relying solely on downstream task accuracy.

6. Significance and Future Outlook

• Solving the Plateau: This work explains why current self-improving LLMs fail (lack of information gain) and provides a roadmap to fix it.
• Beyond RLHF: It suggests that future self-evolving systems cannot rely solely on Reinforcement Learning from Human Feedback (RLHF) or static reward models; they require dynamic, information-theoretic pipelines.
• Call to Action: The authors urge the community to move away from optimizing static loops and toward building dynamic pipelines that explicitly manage capacity, asymmetry, and external information flow.
• Limitations: Currently most applicable to verifiable domains (math, code). Extending this to "hard-to-verify" domains (creative writing, open-ended reasoning) and solving the "knowing what you don't know" problem for proactive seeking remain open challenges.

In summary, the paper argues that sustainable AI self-evolution is impossible without a mechanism to ensure that every iteration of self-synthesis produces data that is strictly more learnable than the previous one, requiring a coordinated expansion of model capacity, task asymmetry, and external context.