Technical Summary: ExpRL: Exploratory RL for LLM Mid-Training

1. Problem Statement

Reinforcement Learning (RL) has become a standard mechanism for improving Large Language Model (LLM) reasoning, particularly for tasks with sparse binary rewards (e.g., final answer correctness). However, the success of sparse-reward RL is critically dependent on the coverage of the base model: the probability mass the model assigns to productive reasoning paths before training begins.

If a base model rarely samples useful intermediate steps or diverse solution strategies for hard problems, sparse rewards provide insufficient learning signals. The model may fail to explore beyond its initial, often suboptimal, reasoning patterns. Current mid-training strategies to address this—such as Supervised Fine-Tuning (SFT) on curated traces or self-distillation—often require manual specification of skills or rely on token-level imitation. These approaches can disrupt the model's native reasoning abilities or fail to generalize to broader problem-solving strategies required for hard combinatorial or mathematical tasks.

The core challenge is to automate the "priming" of a base model to increase its coverage over productive reasoning paths (measured by $pass@k$) without relying on imitation, thereby creating a stronger initialization for subsequent sparse-reward RL.

2. Methodology: Exploratory RL (ExpRL)

ExpRL is an on-policy RL-based mid-training approach that utilizes reference solutions not as targets for imitation, but as reward scaffolds. The method operates on a dataset of hard problems $\mathcal{D}_{mid}$ with associated reference solutions $y^*$.

Core Design Principle

Instead of exposing reference solutions to the policy as demonstrations (which would bias the policy toward imitation), ExpRL hides the references from the policy. The policy samples trajectories $y$ solely from the problem prompt $x$. An LLM judge then compares the sampled trajectory against the hidden reference $y^*$ to assign dense rewards. This leverages the "verification-generation gap," where LLMs are often better at verifying partial progress against a reference than generating a correct solution from scratch.

Reward Mechanisms

ExpRL defines two variants of dense rewards:

1. ExpRL-Outcome: Assigns a dense, graded score to the full rollout based on alignment with the reference solution. The judge evaluates the entire trace against a rubric, providing feedback even if the final answer is incorrect. This rewards partial correctness and useful intermediate reductions.
2. ExpRL-Process: Assigns dense scores to intermediate prefixes of the rollout. To improve credit assignment, the method computes segment-level advantages $A_t$. A segment receives a positive advantage only if it improves the judge's alignment score relative to the previous prefix ($s_t - s_{t-1}$), encouraging the policy to build on intermediate progress and penalizing regression.

Optimization

The policy $\pi_\theta$ is optimized using on-policy RL (GRPO for outcome rewards, REINFORCE for process rewards) with a KL regularization term against the base policy $\pi_0$. The objective maximizes the expected dense reward while staying within a reachable KL space of the base model:
$$\max_\theta \mathbb{E}_{(x, y^*) \sim \mathcal{D}_{mid}} \left[ \mathbb{E}_{y \sim \pi_\theta(\cdot|x)} [R(x, y, y^*)] - \beta \text{KL}(\pi_\theta(\cdot|x) \| \pi_0(\cdot|x)) \right]$$

Downstream Application

After mid-training, the primed policy serves as the initialization for a standard Stage-II sparse-reward RL phase on a downstream dataset $\mathcal{D}'$. In this stage, all reference information is removed, and the model is trained using only binary final-answer rewards.

3. Key Contributions

• Automated Coverage Building: ExpRL proposes a method to automatically broaden the model's coverage over productive reasoning paths using reference solutions as reward scaffolds rather than imitation targets.
• Dense Reward Scaffolding: The paper introduces a mechanism to convert reference solutions into dense, problem-specific grading rubrics via an LLM judge, enabling the reinforcement of partial progress and intermediate reductions that sparse rewards miss.
• Process-Level Credit Assignment: By utilizing prefix-based scoring and segment-level advantages, ExpRL provides granular feedback on the quality of reasoning steps, encouraging behaviors like decomposition and self-correction.
• Empirical Validation of Priming: The work demonstrates that ExpRL produces a superior initialization for downstream sparse-reward RL compared to SFT, sparse-reward GRPO, and self-distillation baselines.

4. Experimental Results

Experiments were conducted on challenging math reasoning tasks (AIME, HMMT, IMO-AnswerBench) and extended to mixed-domain settings (science, coding).

• Downstream Performance: Models initialized with ExpRL achieved higher $pass@1$ and $pass@k$ on held-out benchmarks after Stage-II sparse RL compared to all baselines. Notably, on AIME 2026, ExpRL-Process achieved 63.41% accuracy, outperforming the best baseline (GRPO) at 58.75%.
• Stage-I Improvements: Even before downstream RL, ExpRL-primed models showed significant improvements in $pass@1$ and $pass@16$ on the mid-training distribution, indicating that the method successfully expands the set of solvable problems the model can sample.
• Behavioral Changes: Analysis of reasoning behaviors revealed that ExpRL increases the frequency of search-oriented behaviors such as verification, self-correction, and backtracking relative to the base model. In contrast, SFT tended to reduce verification behaviors.
• Mixed-Domain and Judge Scalability:
  • ExpRL-Outcome successfully improved a larger 8B policy using a smaller 4B judge across math, science, and coding domains, suggesting the judge need not match the policy's scale if it is reference-conditioned.
  • Limitation in Coding: ExpRL showed mixed results on coding tasks (LiveCodeBench). While it improved the base policy, execution-based sparse GRPO remained stronger. The authors attribute this to the fact that coding rewards rely heavily on functional correctness (execution) rather than reference-solution scaffolding, where partial progress is harder to define via a reference path.
• Reward Reliability: Calibration tests confirmed that the dense reward signal depends on the presence of a correct, problem-matched reference. Removing the reference or using a wrong reference significantly degraded the judge's ability to discriminate between correct and incorrect rollouts.

5. Significance and Claims

The paper claims that ExpRL addresses the "exploration bottleneck" in LLM RL by shifting the burden of exploration to the mid-training phase. By rewarding partial progress and diverse reasoning strategies via reference-guided dense rewards, ExpRL builds a stronger "exploration prior."

The authors argue that this approach is superior to imitation-based mid-training (SFT) because it avoids the pitfalls of cloning traces that the base model might not naturally generate, which can lead to distributional shift and instability. Instead, ExpRL keeps the policy on its own distribution while guiding it toward productive regions of the solution space.

The work positions ExpRL as a general priming interface that can be combined with other exploration strategies (e.g., prefix-guided generation) and suggests that the key to scaling RL on hard problems lies in ensuring the base model has sufficient coverage of productive reasoning paths before sparse-reward optimization begins. The paper remains modest, noting that the method requires auxiliary reference data and that its effectiveness varies by domain (e.g., being less critical for coding where execution feedback is available).