🤖 AI

Instruction-Conditioned Exploration with Asymmetric Reinforcement Learning and Self-Distillation

This paper proposes Instruction-Conditioned Exploration (ICE) combined with an Asymmetric Reinforcement Learning and Self-Distillation objective to enhance LLM exploration and transfer diverse behaviors to the test-time policy, achieving significant performance gains in mathematical reasoning tasks.

Original authors: Jim Dilkes, Vahid Yazdanpanah, Sebastian Stein

Published 2026-08-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jim Dilkes, Vahid Yazdanpanah, Sebastian Stein

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to solve a tricky puzzle. You have a powerful tool called "Reinforcement Learning" (RL), which is like a video game coach. The robot tries a move, gets a "good job!" or "try again" signal, and slowly learns to win. But here's the catch: in the world of Large Language Models (LLMs)—the AI brains that write text and solve math problems—the "moves" aren't simple buttons like "jump" or "shoot." They are words. And there are tens of thousands of words to choose from.

If you just tell the robot to "try random things" to learn, it usually just produces gibberish. Randomly swapping a word in a sentence is like trying to fix a car engine by throwing random parts at it; it rarely works. So, the robot tends to get stuck doing the same few things it already knows, missing out on clever new ways to solve problems. The big question for scientists is: How do we get these AI brains to explore new and diverse ways of thinking without just making them say nonsense? This paper tackles that exact problem, proposing a clever way to force the robot to try different "strategies" while it learns, and then teach it how to use those strategies even when the hints are gone.

The "Secret Instruction" Strategy

The researchers, Jim Dilkes and his team from the University of Southampton, came up with a method they call Instruction-Conditioned Exploration (ICE). Think of it like this: Imagine you are training a chess player. Instead of just saying, "Play a game," you give them a different "secret instruction" for every single game they practice. One sheet might say, "Today, focus only on attacking the king's side," while another says, "Defend your pawns at all costs."

In the paper's method, they take a math problem and attach one of several different "behavioral instructions" to it. These aren't the answers; they are just nudge-y hints like, "Look for symmetry in this problem" or "Try breaking it into smaller pieces." By forcing the AI to generate answers while wearing these different "hats," the model explores a much wider variety of solutions than it would on its own. It's like forcing a student to solve the same math problem five different ways just to see which one sticks.

The Teacher and the Student

Here is where the magic happens. The AI model is actually playing two roles at once: a Teacher and a Student.

The Teacher: This version of the AI gets the "secret instructions" (the instructions). It tries to solve the problems using these hints. When it gets a really good answer, it gets a reward.
The Student: This version of the AI is the "real" one we want to use later. It never sees the instructions. It just sees the raw math problem.

The team uses a special training trick called Asymmetric-RL/SD. It's a bit like a master chef (the Teacher) cooking a delicious meal using a secret spice blend. The student (the Student) watches the chef cook, tastes the final dish, and tries to learn how to cook it so well that they can recreate that delicious taste using only basic ingredients, without the secret spice.

The researchers found that by letting the Teacher explore with the hints and then "distilling" (transferring) that knowledge to the Student, the Student becomes much better at solving problems on its own.

What They Found

The team tested this on a specific AI model called Qwen3-1.7B (a "small" model that can run on laptops and phones) solving math problems. They compared their new method against a standard training technique called DAPO.

The Result: When the model was trained with their "Secret Instruction" method (ICE) and the "Teacher-to-Student" transfer (Asymmetric-RL/SD), it got 5.0% better at solving math problems correctly on the first try compared to the standard method.
The Proof: This wasn't a fluke. They ran the experiment five times with different random seeds (like rolling the dice five times), and their method won all 5 times. The improvement was consistent enough that the researchers are quite confident it's a real effect, not just luck.
The Long Game: They also tested this with longer answers (8K context length), and the model still improved, though the boost was a little smaller.

However, there is a limit. When they tried this on a bigger model (4B parameters), the method did not improve the results compared to the standard training. It suggests that this "instruction" trick works best for smaller models that are just on the edge of being able to solve the problem, helping them reach new heights they couldn't get to alone.

Why This Matters

The coolest part is that once the training is done, you don't need the instructions anymore. The "Student" model is ready to go, solving math problems on its own without any extra instructions. This means you can get a smarter, more capable AI on your phone or laptop without needing to carry around a giant list of hints. It's a way to teach the AI to be more creative and thorough during its "schooling" so it becomes a better problem-solver for the rest of its life.

The researchers admit they are still figuring out exactly how far this goes, as it didn't work on the bigger model they tested. But for the smaller, everyday AI models that power our devices, this "Instruction-Conditioned Exploration" looks like a promising new way to help them think outside the box.

Technical Summary: Instruction-Conditioned Exploration with Asymmetric Reinforcement Learning and Self-Distillation

1. Problem Statement

Reinforcement Learning (RL) post-training has become a standard method for enhancing Large Language Models (LLMs), particularly for tasks with verifiable outcomes. However, the authors argue that classical RL exploration methods, which rely on stochastic action selection (e.g., $\epsilon$ -greedy or noise injection), are ill-suited for LLMs. In LLMs, "actions" are tokens drawn from vast vocabularies that must form coherent sequences; random token-level perturbation typically results in incoherent text rather than novel, viable trajectories.

Furthermore, recent studies indicate that RL post-training tends to "sharpen" the model's output distribution, increasing the probability of known correct answers while reducing the coverage of diverse behaviors (support shrinkage). This limits the model's ability to discover new reasoning strategies. Existing exploration methods either operate at the token level (failing to address semantic diversity) or rely on context mediation that lacks explicit mechanisms for transferring train-time behaviors to test-time policies. The core challenge is to induce diverse exploration during training without compromising the performance of the unconditioned policy used at inference.

2. Methodology

The paper proposes a two-part framework: Instruction-Conditioned Exploration (ICE) for training-time diversity and Asymmetric-RL/SD for behavior transfer.

2.1 Instruction-Conditioned Exploration (ICE)

ICE is a template-based approach to "support composition." Instead of relying on random token perturbations, ICE supplements the task prompt $x$ with a behavioral instruction $I$ sampled from a fixed set $\mathcal{I} = \{I_1, \dots, I_N\}$ .

Teacher Policy: During training, the model (acting as a teacher) generates rollouts conditioned on $x + I$ . These instructions provide high-level guidance (e.g., "Ask: what symmetry does this problem have?") to push the model toward distinct problem-solving strategies.
Composite Policy: The training policy becomes a mixture: $\pi^{comp}_\theta(y|x) = \sum_{I \in \mathcal{I}} p(I)\pi_\theta(y|x, I)$ . This broadens the support of the generated text distribution relative to a singular task prompt.
Trade-off: While this increases coverage of strategies, it risks introducing unhelpful instructions that reduce accuracy.

2.2 Asymmetric-RL/SD

To transfer the behaviors learned by the instruction-conditioned teacher to the unconditioned test-time policy, the authors introduce Asymmetric-RL/SD, a self-distillation objective combining Reinforcement Learning and Knowledge Distillation.

Teacher ( $\pi^T_\theta$ ): The instruction-conditioned policy.
Student ( $\pi^S_\theta$ ): The unconditioned policy (used at test time), sharing parameters $\theta$ with the teacher.
Objective Function: The training objective $J(\theta)$ $J (θ)$ combines:
1. RL Term: A reward-maximizing term for the teacher policy (using the DAPO variant of GRPO).
2. Distillation Term: A forward-KL divergence term ( $J^{KL}_S$ ) that pulls the student policy toward the teacher's correct rollouts.
Asymmetry: Unlike standard self-distillation which often balances rewards for both policies, Asymmetric-RL/SD sets the student reward weight $\alpha=1$ (teacher-only reward maximization) and applies the KL term only to the student, filtering teacher rollouts to include only those with correct answers ( $R=1$ ). This ensures the student learns to replicate useful, verified behaviors discovered by the teacher without being penalized for the teacher's exploration of incorrect paths.

3. Key Contributions

Instruction-Conditioned Exploration (ICE): A novel method for inducing diverse rollouts during RL post-training by conditioning generation on a set of distinct, high-level behavioral instructions, thereby expanding the support of the policy during training.
Asymmetric-RL/SD Objective: A new training objective that combines RL on a teacher policy with forward-KL self-distillation to an unconditioned student. This explicitly transfers the benefits of instruction-conditioned exploration to the test-time policy.
Empirical Validation: A multi-seed study across different model scales (1.7B, 4B) and context lengths (4K, 8K) demonstrating the efficacy of the approach on mathematical reasoning tasks.

4. Results

The experiments were conducted using Qwen3 models on mathematical reasoning benchmarks (AIME, HMMT, MATH-500).

1.7B / 4K Setting:
- ICE with Asymmetric-RL/SD improved the held-out pass@1 by 5.0% relative to the DAPO baseline.
- This configuration achieved 5/5 benchmark wins with a 95% bootstrap confidence interval excluding zero.
- The improvement over using RL alone with instructions (without the distillation term) was smaller and less certain, suggesting the explicit distillation signal is crucial for transfer.
- Training convergence was not slowed by the inclusion of diverse instructions.
1.7B / 8K Setting:
- A smaller but persistent improvement of 3.5% (3/5 wins) was observed, indicating the method generalizes to longer contexts, though the gain is weaker.
4B / 4K Setting:
- The method did not improve over DAPO (resulting in a -3.3% change). The authors note that the 4B base model may already possess sufficient capability, leaving less "headroom" for exploratory strategies to provide value compared to the 1.7B model.
Coverage vs. Accuracy:
- While RL post-training typically increases pass@1 but decreases pass@k (coverage) at high $k$ , Asymmetric-RL/SD improved pass@1 over DAPO while maintaining comparable coverage.
- Sampling with the behavioral instructions at inference (even for the base model) recovered high coverage, suggesting the instructions themselves surface correct solutions independently of the RL process.

5. Significance and Claims

The paper claims that instruction-conditioned exploration, paired with an explicit self-distillation transfer mechanism, can improve the held-out reasoning capabilities of small language models (specifically at the 1.7B scale) beyond standard RL baselines like DAPO.

Key implications highlighted by the authors include:

Cost Efficiency: The method adds no inference cost; the test-time policy is unconditioned and standard. The gains are achieved entirely through a modified training process.
Small Model Utility: Improving small models is critical for deployment on consumer devices (phones, laptops) where larger models are infeasible.
Mechanism of Action: The results suggest that parameter sharing alone is insufficient to transfer teacher behaviors; the explicit forward-KL distillation signal is necessary to incorporate the teacher's discovered strategies into the student policy.

The authors remain modest regarding the scope, noting that the gains did not transfer to the 4B model under the tested conditions and that the study is limited to mathematical reasoning with binary rewards. They position the work as a step toward characterizing where such gains hold across different model scales and contexts.