Technical Summary: INFUSER – Influence-Guided Self-Evolution Improves Reasoning

1. Problem Statement

Recent progress in large language model (LLM) reasoning relies heavily on Reinforcement Learning with Verifiable Rewards (RLVR). However, the scalability of RLVR is bottlenecked by the scarcity of high-quality, verifiable training data, which is costly to curate. Self-evolution offers a path forward by allowing a model to generate its own training data from unstructured documents or its own outputs, eliminating the need for external teacher models or curated corpora.

Existing self-evolution methods face two primary limitations:

1. Grounding: Pure self-play methods (e.g., R-Zero) rely solely on the model's prior knowledge, risking hallucinated reference answers. Executor-based methods (e.g., AZR) restore verifiability but are restricted to domains with formal executors (code, math), failing to leverage abundant unstructured text.
2. Generator Objectives: Document-grounded approaches (e.g., SPICE) often reward the generator based on difficulty heuristics (e.g., maximizing the solver's failure rate to ~50%). Difficulty is a coarse proxy for utility; a question may be "difficult" due to ambiguity or misalignment with the source, offering no guarantee of improving the solver on the target distribution.

The core question addressed is: Can we train a generator to produce document-grounded training data that genuinely improves the current solver on a target distribution, while co-evolving with it?

2. Methodology: INFUSER

The authors propose INFUSER (INFluence-gUided Self-Evolution Improves Reasoning), an iterative co-training framework that casts self-evolution as a bilevel game between a Generator ($\pi_\phi$) and a Solver ($\pi_\theta$). Both are initialized from the same pretrained model.

2.1 Bilevel Game Formulation

• Lower Level (Solver): The solver trains on a curriculum of question-answer (QA) pairs ($Q_\phi$) generated by the generator from a pool of unstructured documents ($D_{doc}$). The solver optimizes to maximize accuracy on this curriculum using standard RLVR (e.g., Dr.GRPO).
• Upper Level (Generator): The generator aims to produce a curriculum $Q_\phi$ such that, after the solver updates, the solver's performance improves on a target distribution ($P$).
• Cooperative Nature: Unlike adversarial self-play, the generator is rewarded for helping the solver improve on the target distribution, not for defeating it.

2.2 Optimizer-Aware Influence Score

To avoid the prohibitive cost of exact bilevel differentiation (backpropagating through the solver's full training loop), INFUSER employs a first-order approximation inspired by influence functions.

• Dev-Set Anchor: A small, held-out development set ($D_{dev}$) sampled from the target distribution $P$ is used to define the "improvement direction." The gradient of the solver's performance on $D_{dev}$ ($\hat{g}_{dev}$) serves as the target vector.
• Influence Calculation: For each generated QA pair $(q, a_\phi)$, the system estimates the solver's parameter update direction ($\hat{\Gamma}(q, a_\phi)$) induced by training on that specific pair. This direction is preconditioned by the AdamW optimizer state to reflect the actual update step.
• Reward Signal: The generator's reward is the cosine similarity between the target gradient ($\hat{g}_{dev}$) and the induced update direction ($\hat{\Gamma}$):
  $$s(q, a_\phi) = \text{cossim}(\hat{g}_{dev}, \hat{\Gamma}(q, a_\phi))$$
  A positive score indicates that training on $(q, a_\phi)$ aligns with improving performance on the target distribution. This score replaces the traditional "difficulty" metric.

2.3 DuGRPO: Dual-Normalized GRPO

The influence score is a continuous, noisy signal, which poses challenges for standard Group Relative Policy Optimization (GRPO).

• Challenge: Normalizing rewards solely within a document group (GRPO-style) amplifies noise in low-variance groups. Normalizing solely across the batch (BatchNorm-style) allows high-variance groups to dominate.
• Solution: The authors propose DuGRPO, a dual-normalized advantage estimator that combines within-group and cross-group normalization:
  $$\hat{A}_{gen} = \frac{s(q, a_\phi) - \mu_d}{\sigma_d + \sigma_B + \epsilon}$$
  where $\sigma_d$ is the standard deviation within the document group and $\sigma_B$ is the mean standard deviation across the batch. This stabilizes training by damping low-variance noise while preventing high-variance dominance.

2.4 Training Loop

The algorithm proceeds in phases per iteration:

1. Dev Reference: Compute the target gradient $\hat{g}_{dev}$ from the solver's rollouts on $D_{dev}$.
2. Generation: The generator samples documents and produces QA pairs.
3. Solver Evaluation: The solver attempts the generated questions; per-question gradients are estimated.
4. Influence Scoring: Compute influence scores using the cosine similarity with $\hat{g}_{dev}$.
5. Updates: Update the generator using DuGRPO with influence scores as rewards; update the solver using Dr.GRPO with binary correctness rewards.

3. Key Contributions

1. Influence-Guided Self-Evolution: A novel framework that replaces difficulty-based generator rewards with an optimizer-aware influence score, directly optimizing for utility on a target distribution rather than just difficulty.
2. DuGRPO: A new policy gradient variant designed specifically for the continuous, noisy nature of influence-based rewards, enabling stable generator training.
3. Document-Grounded Co-Evolution: A method that leverages unstructured textbooks to generate training data, avoiding the need for external teachers or executors while maintaining verifiability through the solver's own reasoning.
4. Scalability: Demonstrating that a co-evolving 8B generator can outperform a frozen 32B "thinking" generator on math and coding tasks, suggesting that adaptation is more critical than raw model scale for curriculum generation.

4. Experimental Results

Experiments were conducted on Qwen3-4B-Base and Qwen3-8B-Base anchors, comparing INFUSER against strong baselines (R-Zero, AZR, R-Few, SPICE) and a General-Reasoner baseline.

• Performance Gains: INFUSER achieved the highest scores on 8 of 14 benchmarks. On Qwen3-8B-Base, it showed relative improvements of over 20% on OlympiadBench (Math), SuperGPQA, and GPQA-Diamond compared to the base model.
• Scaling: The performance gains of INFUSER were consistent across model sizes (4B to 8B), whereas baselines like R-Zero and SPICE showed significant performance decay when scaling up.
• Generator Quality: Analysis showed that the co-evolving generator produces increasingly well-posed and challenging questions over time. The "Strong-vs-Weak" gap (accuracy difference between a strong model and the base model on generated questions) increased, indicating the generator is creating questions that are hard for the base model but solvable by stronger models (i.e., high-quality reasoning challenges).
• Ablation Studies:
  • Generator Necessity: Removing the generator (training directly on $D_{dev}$) led to overfitting on the dev set and poor generalization.
  • Co-evolution: A frozen 8B generator underperformed the co-evolving 8B generator, especially in math. A frozen 32B generator outperformed the 8B co-evolving generator on knowledge-heavy tasks (general reasoning, medical) but was outperformed on math and coding, highlighting that adaptation beats scale for reasoning.
  • Components: Both the optimizer-aware influence score and DuGRPO normalization were found to be essential; removing them caused performance to regress to the level of a frozen generator.
• Extensions:
  • Instruction-Finetuned Anchors: INFUSER successfully improved an instruction-finetuned anchor (OLMo-3-7B-Instruct-SFT), leading on 10 of 13 benchmarks.
  • Hybrid RLVR: Combining document-grounded self-evolution with rule-verifiable math RLVR stabilized the "response length collapse" observed in science-only training, reducing seed variance in math performance.

5. Significance and Claims

The paper claims that INFUSER offers a scalable path to stronger reasoning by converting abundant unstructured corpora into structured, utility-aligned training signals without relying on curated datasets or teacher models.

• Shift in Paradigm: It moves the generator's objective from "making things hard" to "making things useful," using influence functions to align curriculum generation with the solver's learning trajectory.
• Efficiency: It demonstrates that a smaller, co-evolving generator can outperform a much larger, frozen generator on reasoning tasks, suggesting that the process of curriculum adaptation is more valuable than the static knowledge of a larger model.
• Generalizability: The framework is shown to be flexible, working with both base and instruction-finetuned models, and capable of integrating with other RLVR signals (like math verifiers) to stabilize training dynamics.

The authors conclude that while the framework currently relies on a small dev set to anchor the target direction, it represents a promising direction for scaling reasoning capabilities by coupling utility-based curriculum generation with larger document pools and longer training horizons.