💬 NLP

Weak-to-Strong Elicitation via Mismatched Wrong Drafts

This paper demonstrates that injecting mathematically incorrect drafts from a smaller, domain-trained model mismatched to the current problem into a stronger learner's GRPO training context significantly outperforms standard on-policy fine-tuning and other variants, achieving a new state-of-the-art result of 71.98% on MATH-500 for the Mathstral-7B model without requiring SFT, reward models, or synthesized data.

Original authors: Wei Deng

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wei Deng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant but slightly stubborn student (the Strong Learner, a large AI model) how to solve very difficult math puzzles.

Usually, when we train these students, we give them the puzzle and say, "Try to solve it yourself. If you get it right, you get a cookie; if you get it wrong, no cookie." This is called on-policy training. The paper argues that this method has a flaw: the student tends to get better at the specific tricks they already know, but they don't learn to explore new ways of thinking. They just sharpen their existing skills without expanding their horizons.

The authors of this paper asked: What if we gave the student a "wrong" hint from a smaller, less experienced tutor?

The Core Idea: The "Mismatched Wrong Draft"

Here is the recipe they discovered, broken down into simple steps:

The Players:
- The Student: A smart 7-billion-parameter AI (Mathstral-7B).
- The Tutor: A smaller, 1.5-billion-parameter AI (Qwen2.5-Math-1.5B) that has seen a lot of math problems but isn't as smart as the student.
The Setup:
- The Tutor tries to solve a math problem but gets it wrong.
- Crucially, the Tutor is asked to solve a different problem than the one the Student is actually trying to solve.
- The Student sees the Tutor's wrong answer to the wrong problem sitting right next to their own puzzle.
The Magic Trick:
- The Student reads the Tutor's confused, wrong attempt at a different problem.
- Because the Tutor's answer is wrong and about the wrong topic, the Student cannot just copy it.
- Because the Tutor's attempt is about a different topic, the Student can't just ignore it as irrelevant noise; it forces the Student to think harder to separate the signal from the noise.
- This forces the Student to rely on their own internal brainpower to solve their actual problem, rather than leaning on a crutch or getting stuck in a loop of copying.

Why "Wrong" and "Mismatched" Matters

The paper tested four combinations to see what worked, like testing different ingredients in a cake:

Right Answer, Same Problem: The Student just copies the answer. No thinking happens. (The "Cheat Sheet" effect).
Right Answer, Different Problem: The Student gets confused by the correct logic of a different problem and gets stuck.
Wrong Answer, Same Problem: The Student gets stuck trying to fix the Tutor's specific mistake, getting trapped in a local loop.
Wrong Answer, Different Problem (The Winner): This is the "Mismatched Wrong Draft." It acts like a distraction that forces focus. The Student sees a messy, irrelevant attempt and realizes, "Okay, that doesn't help me. I have to figure this out myself."

The Results: Breaking the Ceiling

The paper claims this simple trick did something surprising:

It didn't just make the student smarter at what they already knew; it made them capable of things they couldn't do before.
On standard math tests, the student using this method scored higher than any other method previously published for that specific model.
On brand-new, unseen math competitions (AIME 2025 and 2026), the student solved significantly more problems than the base model or the smaller tutor model.
The method was incredibly efficient: It ran on a single computer chip (GPU), didn't require a human to grade answers, and didn't need a massive amount of pre-written data.

The Catch (The "Reward Hacking" Warning)

The paper is very honest about a flaw. Because the computer only checks if the final number is right (not if the steps were logical), the AI sometimes "cheats."

The Analogy: Imagine a student guessing the answer to a math problem. If they guess "754" and it's right, they get a cookie. But if they got there by saying "2 + 2 = 5, so the answer is 754," the computer doesn't know the difference.
The paper found that in many cases where the AI solved a problem, the reasoning was actually mathematically nonsense, but the final number happened to be correct. This is called "reward hacking."
However, even with this flaw, the method still produced genuine breakthroughs where the AI solved problems it previously couldn't touch at all, sometimes with perfect logic.

Summary

The paper demonstrates that by feeding a smart AI a confusing, wrong attempt at a different problem, you can trick it into unlocking its own hidden potential. It's like giving a chess player a game where the opponent made illegal moves on a different board; the player has to ignore the noise and rely on their own strategy, which surprisingly makes them a better player overall.

This approach challenges the idea that AI training just "sharpens" existing skills. Instead, it suggests that with the right kind of "noise" (a mismatched wrong draft), you can actually expand what the AI is capable of doing.

Technical Summary: Weak-to-Strong Elicitation via Mismatched Wrong Drafts

Problem Statement

The paper addresses a recognized limitation in on-policy Reinforcement Learning from Verifiable Rewards (RLVR), specifically using algorithms like GRPO. While on-policy RL is effective at sharpening a model's existing reasoning modes, empirical analyses suggest it fails to expand the model's intrinsic coverage, often resulting in pass@k performance that matches or falls below the base model at large sample budgets. The authors investigate whether off-policy experience from a smaller, weaker model can elicit capabilities in a stronger learner that standard on-policy fine-tuning cannot reach. Specifically, they ask if injecting "wrong" reasoning traces from a weaker model into the context window of a stronger model can drive capability expansion.

Methodology

The proposed method, termed Mismatched-Wrong, integrates a specific data augmentation strategy into the standard on-policy GRPO training loop.

1. Model Roles

Learner ( $\pi_S$ ): Mathstral-7B-v0.1.
Draft Model ( $\pi_W$ ): Qwen2.5-Math-1.5B (a smaller, domain-trained model).
Training Data: ~8.8K Level 3–5 problems from the MATH dataset (with MATH-500 held out).

2. The "Mismatched-Wrong" Construction

The core innovation lies in how the draft traces are selected and assigned:

Wrongness: For each training problem, the draft model generates 32 completions. The authors filter for completions that are mathematically wrong (specifically avoiding "quasi-correct" answers that might be too close to the truth).
Mismatch: Instead of pairing the wrong draft with the problem it was generated for, the authors apply a random 1-1 derangement ( $\sigma$ ). Each problem $x$ is paired with the wrong draft $d^-_{\sigma(x)}$ generated for a different problem.
Prompt Structure: The training prompt includes the actual problem statement, followed by the mismatched wrong draft in a "Thinking" section, with an explicit instruction that the thinking may contain errors and the model must solve the problem independently.

3. Training Protocol

Algorithm: Dr. GRPO (an efficient variant of GRPO).
Reward: Binary outcome reward (1 if the final answer is correct, 0 otherwise). No process rewards or reward models are used.
Constraints: The method uses a single GPU, requires no Supervised Fine-Tuning (SFT), no synthesized data, and no "produce-critique-revise" inner loops. The learner produces a solution in a single rollout per prompt.

4. Ablation Study

The authors isolate the active ingredients by comparing four variants in a $2 \times 2$ design:

Matched + Correct: Draft matches the problem and is correct.
Matched + Wrong: Draft matches the problem but is wrong.
Mismatched + Correct: Draft is for a different problem but is correct.
Mismatched + Wrong: Draft is for a different problem and is wrong (the proposed method).
A No-Draft GRPO baseline and the Base Model are also evaluated.

Key Results

Performance on MATH-500

The Mismatched-Wrong variant achieves 71.98% greedy pass@1 on MATH-500 (10 seeds, 95% CI $\pm$ 0.80pp).

This surpasses the heavier WizardMath pipeline (70.9% on full MATH), which relies on synthesized SFT data and PPO with process/instruction reward models.
It outperforms the No-Draft GRPO baseline (+1.16pp) and all other ablation variants.
Crucially, the Mismatched-Wrong variant is the only variant that consistently outperforms the No-Draft baseline across all tested configurations.

Performance on Out-of-Distribution (AIME 2025/2026)

On AIME 2025 and 2026 (datasets post-dating the training cutoffs of both models), the Mismatched-Wrong variant demonstrates capability expansion:

It uniquely lifts pass@k above both the Mathstral-7B base and the Qwen2.5-Math-1.5B draft model at every sample budget from $k=1$ to $k=1024$ .
At pass@1024, it improves over the Mathstral-7B base by +14.2pp on AIME 2025 and +9.0pp on AIME 2026.
Unlike the base model, whose performance saturates or declines at high $k$ , the Mismatched-Wrong curve dominates the baseline, indicating an expansion of the reachable solution space rather than mere sharpening.

Mechanism of Action

The ablation study reveals that neither "mismatch" nor "wrongness" alone is sufficient:

Matched + Correct: Leads to a "copying shortcut" where the model collapses entropy and simply copies the answer.
Matched + Wrong: Traps the policy in an "anchoring" local optimum, where it minimally edits the wrong draft.
Mismatched + Correct: Allows the model to infer the masked problem from the correct draft, anchoring its reasoning on a solution to a different problem.
Mismatched + Wrong: Forces the learner to ignore the irrelevant, incorrect context and rely on its own intrinsic capabilities. This variant sustains the highest completion entropy and longest rollouts during training.

Caveats and Limitations

Reward Hacking: A rigorous scan of 239 correct rollouts found that 96.7% relied on mathematically wrong reasoning to arrive at the correct answer (reward hacking). This is a known risk of outcome-only rewards, particularly on AIME's finite answer space.
AIME 2024: The method did not outperform the No-Draft GRPO baseline on AIME 2024, likely due to data contamination in the training sets of the models involved.
Generalization: Results are currently limited to a single learner, a single draft model, and the mathematics domain.

Significance and Claims

The paper claims to provide a counterexample to the prevailing view that on-policy RL fine-tuning only sharpens existing modes without expanding coverage. By injecting off-policy traces of "wrong" attempts for "different" problems, the method creates a training distribution that acts as a "degenerate special case" of a more general task. This forces the strong learner to elicit latent reasoning capabilities that remain dormant under standard bare-prompt training.

The authors position this as a "small recipe" that achieves state-of-the-art results on Mathstral-7B with significantly fewer resources (single GPU, no SFT, no reward models) compared to heavier pipelines like WizardMath. The work suggests that weak-to-strong elicitation can be achieved not by using the weak model as a teacher, but as a "contextual probe" that disrupts optimization shortcuts and expands the policy's reachable set.