Technical Summary: Confidence-Orchestrated Self-Evolution (COSE)

Problem Statement

Self-evolving Large Language Models (LLMs) aim to improve reasoning capabilities by generating their own training tasks, solving them, and learning from self-generated feedback without relying entirely on human-curated supervision. However, a critical bottleneck exists in domains where solutions cannot be verified by execution (e.g., code) or exact matching (e.g., mathematics). In these "open-domain" settings, the model must act as both a Validator (judging task validity) and a Judge (assessing answer correctness).

The core challenge is noisy self-generated feedback. A model capable of generating difficult tasks may lack the capability to correctly validate them or judge the correctness of proposed solutions. When the model erroneously endorses a flawed task or incorrectly judges a wrong answer as correct, these errors are converted into reward signals. In standard self-evolution loops, these erroneous signals drive gradient updates that reinforce incorrect reasoning patterns, leading to optimization instability or performance degradation. Existing approaches either rely on external verifiers (limiting generality) or treat all self-generated feedback as equally trustworthy supervision, failing to account for the uncertainty inherent in the feedback generation process.

Methodology: COSE

The authors propose COSE (Confidence-Orchestrated Self-Evolution), a framework that uses the LLM's intrinsic confidence as a lightweight uncertainty signal to modulate the learning process. COSE employs a single backbone model $\pi_\theta$ prompted to play four roles: Proposer (generates questions), Validator (scores question quality), Solver (generates answers), and Judge (scores answer correctness).

The core innovation lies in estimating confidence directly from the token-level output distribution of the Validator and Judge during feedback generation, rather than treating the feedback as a ground-truth label.

1. Confidence Estimation

COSE calculates confidence ($c$) based on the normalized entropy of the model's token-level output distribution. For a generated token $t$ with probability distribution $P_t$ over the vocabulary $\mathcal{V}$:
$$c_t = 1 - \frac{H(P_t)}{\log |\mathcal{V}|}$$
Sequence-level confidence ($c_{seq}$) is the average of token-level confidences. This metric serves as a heuristic for uncertainty: low entropy (high confidence) implies the model is certain about its judgment, while high entropy (low confidence) indicates uncertainty.

2. Confidence-Weighted PPO

COSE integrates these confidence scores into the Proximal Policy Optimization (PPO) objective to scale the gradient contribution of each sample.

• Proposer Weight ($w_P$): Scales updates based on question quality ($v$) and Validator confidence ($c_V$):
  $$w_P(q) = \text{clip}(v(q) \cdot c_V(q), 0.1, 1.0)$$
• Solver Weight ($w_S$): Scales updates based on question quality, Validator confidence, and Judge confidence ($c_J$):
  $$w_S(q, a) = \text{clip}(v(q) \cdot c_V(q) \cdot c_J(q, a), 0.1, 1.0)$$
  By capping the weight at a lower bound of 0.1 rather than discarding uncertain samples entirely, COSE preserves exploration while preventing high-variance, low-confidence feedback from dominating the optimization.

3. Confidence-Prioritized Replay

The replay buffer prioritizes sampling questions that are not only valid but also lie near the model's current capability boundary. The priority is defined as:
$$\text{priority}(q) \propto v(q) \cdot c_V(q) \cdot \max(4p(q)(1-p(q)), 0.1)$$
Here, $p(q)$ is the average solve rate. The term $4p(q)(1-p(q))$ is maximized when $p(q) \approx 0.5$, focusing training on questions the model can solve roughly half the time (the "learning frontier"). The confidence terms ensure that questions with uncertain validation are not over-sampled.

Key Contributions

1. Identification of Optimization Bottleneck: The paper identifies noisy self-generated feedback (both validation and judgment) as a central optimization bottleneck in self-evolving LLMs, distinct from mere data scarcity.
2. Formulation of Training Control: The authors formulate the problem as a per-sample training-control task, where the influence of a feedback signal on learning is modulated by its intrinsic confidence.
3. Empirical Validation: Through extensive experiments across 19 held-out benchmarks and four model backbones (Qwen and Llama families, 0.6B–4B parameters), the authors demonstrate that COSE consistently improves over base models and outperforms or remains competitive with prior self-evolving baselines (AZR, MAE, R-Zero).

Experimental Results

The evaluation covers general reasoning, mathematics, and code generation.

• General Reasoning: COSE achieves the highest average performance across all four backbones. Notably, on the weaker Qwen3-0.6B and Llama-3.2-3B models, COSE yields substantial gains (e.g., +28.5 points on Qwen3-0.6B), significantly outperforming baselines. The authors attribute this to the fact that weaker models produce noisier feedback; COSE's confidence weighting effectively dampens the impact of these errors, whereas baselines reinforce them.
• Mathematics: COSE shows the largest improvements on difficult mathematical benchmarks (e.g., Olympiad-Bench, Minerva). The results suggest that confidence control is particularly vital when the Judge must distinguish between plausible but incorrect derivations, a scenario where naive self-judging often fails.
• Code Generation: In domains with executable verification (code), COSE remains competitive. While methods relying on external verifiers (like AZR) sometimes perform slightly better on specific backbones, COSE does not suffer significant regression. This indicates that confidence weighting does not harm performance in domains where feedback is already reliable.
• Ablation Studies: Removing confidence-weighted PPO causes the most significant performance drop, particularly in mathematics, confirming that controlling gradient strength is the primary driver of success. Removing replay priority has a smaller but consistent negative effect.

Significance and Claims

The paper claims that self-evolution becomes more effective when models not only learn from their own feedback but also control how strongly uncertain feedback affects training.

• Robustness: COSE provides a mechanism to prevent the amplification of errors in self-evolution loops. It allows models to benefit from self-generated data even when the Validator/Judge is imperfect, provided the model can recognize its own uncertainty.
• Generalizability: Unlike methods restricted to executable domains, COSE enables self-evolution in open-domain reasoning tasks where external verifiers are unavailable.
• Modesty: The authors explicitly state that confidence is a heuristic uncertainty signal, not a correctness guarantee. They acknowledge that high-confidence outputs can still be wrong, particularly on adversarial or out-of-distribution problems. Furthermore, the method relies on access to token-level probabilities (logits), limiting its direct application to black-box APIs without additional sampling strategies. The experiments are conducted on relatively small open-weight models (0.6B–4B), and the scaling behavior to larger models remains an open question for future work.

In summary, COSE represents a shift from treating self-generated feedback as uniform supervision to treating it as a weighted signal, where the weight is determined by the model's own confidence in producing that signal. This approach stabilizes the self-evolution process, particularly in the challenging intermediate regime where models are improving but their self-evaluation capabilities are still noisy.