💬 NLP

QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation

The paper introduces QuestA, a reinforcement learning strategy that enhances large language model reasoning by augmenting training questions with partial solutions, achieving new state-of-the-art results on challenging math benchmarks even with 1.5B-parameter models.

Original authors: Jiazheng Li, Hongzhou Lin, Hong Lu, Kaiyue Wen, Zaiwen Yang, Jiaxuan Gao, Yi Wu, Jingzhao Zhang

Published 2026-04-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiazheng Li, Hongzhou Lin, Hong Lu, Kaiyue Wen, Zaiwen Yang, Jiaxuan Gao, Yi Wu, Jingzhao Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Too Hard" Wall

Imagine you are trying to teach a student (a Large Language Model) how to solve incredibly difficult math problems, like those found in the world's toughest math competitions.

You have two main ways to teach them:

Give them easy problems: They get everything right quickly, but they stop thinking deeply. They just memorize patterns. When you finally give them a hard problem, they crash because they never learned how to struggle or explore new ideas.
Give them hard problems: This is where they need to learn. But here's the catch: if the problem is too hard, the student gets stuck immediately. They guess randomly, get it wrong, and get no feedback. It's like trying to find a needle in a haystack blindfolded. The teacher (the training algorithm) can't help because the student never gets a "correct" answer to reinforce. They just spin their wheels, wasting time and energy.

This is the dilemma researchers faced: Easy problems make the model lazy; hard problems make the model give up.

The Solution: QUESTA (The "Training Wheels" Strategy)

The authors of this paper, QUESTA, came up with a clever trick to fix this. They realized that instead of throwing the student into the deep end, they should give them partial solutions or hints right at the start.

Think of it like this:

The Old Way: You hand a student a complex physics equation and say, "Solve this!" If they don't know the first step, they fail.
The QUESTA Way: You hand them the same equation, but you write the first half of the solution on the board for them. You say, "Here is the setup. Now, you figure out the rest."

By giving the model a "head start" (a partial solution), the problem becomes manageable. The model can successfully finish the problem, get a "Good Job!" reward, and learn how to think.

How It Works in Practice

The researchers took a dataset of 220,000 math problems. They filtered out the easy ones (because they aren't useful for growth) and the impossible ones (where the model fails 100% of the time).

Then, they created a special training curriculum:

Phase 1 (Heavy Hints): They took the hardest problems and gave the model 50% of the answer as a hint. The model learned to finish the job.
Phase 2 (Light Hints): Once the model got good at that, they reduced the hint to just 25%. The model had to do more of the work, but it still had a safety net.

This is like a video game where you start on "Easy Mode" with a guide, but as you level up, the guide disappears, forcing you to play on "Hard Mode" using the skills you just learned.

The Magic Result: Small Models, Big Brains

The most exciting part of this paper is the result. They tested this on a very small model (only 1.5 billion parameters). Usually, small models are considered "dumb" compared to giant models with 32 billion or more parameters.

However, using QUESTA:

The tiny 1.5B model beat the massive 32B models on several difficult math benchmarks.
It solved problems that the giant models couldn't solve.
It didn't just get better at one type of problem; it became a better reasoner overall.

Why This Matters (The "Aha!" Moment)

Usually, when you train AI on hard problems, it tends to "collapse." It gets so confident in one specific way of solving things that it stops exploring. It becomes rigid.

QUESTA prevents this collapse. Because the model is constantly being challenged with just the right amount of difficulty (thanks to the hints), it keeps its curiosity alive. It learns to explore different paths to the solution, not just the one it memorized.

Summary Analogy

Imagine you are trying to climb a massive, sheer mountain (Reasoning Capacity).

Standard Training: You either start at the bottom on a flat path (easy problems) and never reach the peak, or you try to jump straight to the top (hard problems) and fall off.
QUESTA: You build a series of scaffolding platforms up the side of the mountain.
- First, you build a platform halfway up and let the climber practice climbing from there.
- Once they are strong, you remove the bottom half of the scaffolding, forcing them to climb the lower section themselves.
- Eventually, the scaffolding is gone, but the climber is strong enough to reach the summit on their own.

The Bottom Line: QUESTA proves that you don't need a bigger brain to solve harder problems; you just need a smarter way to teach it. By giving AI "partial answers" during training, we can turn small, cheap models into reasoning geniuses.

1. Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has become a dominant paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs). However, recent studies suggest a critical limitation: standard RL often fails to expand a model's intrinsic reasoning capacity beyond what is already encoded in the base model. Instead, it tends to exploit existing knowledge or overfit to shallow patterns.

The paper identifies a fundamental trade-off in RL training:

Easy Prompts: Training on problems the model can already solve leads to "entropy collapse," where the model overfits to known solution modes, reducing diversity and degrading performance on harder tasks (lowering pass@k as $k$ increases).
Hard Prompts: Training directly on difficult problems where the base model has near-zero success rates leads to extremely sparse reward signals. This results in slow learning, sample inefficiency, and often a complete failure to make progress because the model cannot generate any correct trajectories to receive a reward.

Core Challenge: How to structure RL training to tackle high-difficulty reasoning problems effectively without suffering from sparse rewards or entropy collapse?

2. Methodology: QUESTA

The authors propose QUESTA (Question Augmentation), a data-centric strategy that injects partial solutions (hints) into the training prompts to scaffold the learning process.

Key Mechanisms:

Partial Solution Injection: Instead of presenting a raw hard problem, QUESTA prepends a portion of the ground-truth solution (e.g., the first 50% of the reasoning steps or solution block) to the prompt. This transforms an intractable problem into a tractable one where the model only needs to complete the remaining steps.
Curriculum Learning (Two-Stage):
1. Stage 1 (High Hint): The model is trained on augmented prompts with a high percentage of hints (e.g., $p=50\%$ ). This provides dense reward signals, allowing the model to learn the structure of complex solutions.
2. Stage 2 (Reduced Hint): The hint percentage is reduced (e.g., $p=25\%$ ). The model is forced to generate more of the solution autonomously, gradually transitioning from scaffolded reasoning to independent reasoning.
Difficulty Filtering: The method specifically targets the "hardest" subset of the dataset (where the base model's pass rate is near zero). It uses a two-stage filtering pipeline to ensure augmentation resources are concentrated on problems that genuinely require scaffolding.
Orthogonality: QUESTA operates at the input level and is compatible with existing RL algorithms (e.g., GRPO, DAPO) without modifying the reward function or optimization algorithms.

Theoretical Insight:

The paper provides a theoretical analysis showing that without hints, if the model's "capacity set" (likely outputs) does not intersect with the "solution set," the probability of sampling a correct trajectory is vanishingly small ( $O(1/\delta_p)$ ). By providing a hint, the method effectively decomposes the problem, increasing the probability of sampling a valid trajectory to $O(1/\sqrt{\delta_p})$ . This significantly improves sample efficiency and allows the model to discover solution paths it would otherwise miss.

3. Key Contributions

Empirical Discovery: The authors demonstrate that training RL on easy prompts degrades pass@k performance (reducing reasoning diversity), while training on hard prompts without scaffolding is inefficient.
QUESTA Framework: Introduction of a simple, plug-and-play data augmentation method that uses partial solutions to create a smooth curriculum for RL training.
Theoretical Justification: A formal proof showing that partial-solution augmentation reduces the sampling budget required to find a solution from $O(1/\delta_p)$ to $O(1/\sqrt{\delta_p})$ , explaining the acceleration in learning.
State-of-the-Art (SOTA) Results: Achieving SOTA performance on mathematical benchmarks using a 1.5B-parameter model, outperforming significantly larger models (e.g., 32B parameters) in specific contexts.

4. Experimental Results

The authors evaluated QUESTA on the Nemotron-1.5B and DeepScaleR-1.5B models using the OpenR1-Math-220K dataset.

Performance Benchmarks (Pass@1, Avg@32):

QUESTA-Nemotron-1.5B achieved new SOTA results for 1.5B models, surpassing much larger models:

AIME24: 72.50% (vs. 61.77% baseline; +10.73% gain).
AIME25: 62.29% (vs. 49.50% baseline; +12.79% gain).
HMMT25: 41.67% (vs. 31.56% baseline; +10.11% gain).
Comparison: The 1.5B QUESTA model outperformed DeepSeek-R1-Distill-32B on AIME25 by +11% and matched or exceeded it on other benchmarks, despite having ~20x fewer parameters.

Key Observations:

Pass@k Preservation: Unlike standard RL which often sees pass@k drop as $k$ increases (due to overfitting), QUESTA maintains or improves pass@k across all $k$ , indicating enhanced solution diversity.
Entropy Stability: Training with QUESTA prevents entropy collapse. The model maintains exploratory behavior throughout training.
Generalization: The improvements generalize to problems without hints at evaluation time. The model learns to solve the full problem autonomously after being trained with scaffolding.
Efficiency: The curriculum approach (50% $\to$ 25% hints) converges faster and achieves higher accuracy than training with only high-hint or no-hint data.

5. Significance

Democratizing Reasoning: QUESTA demonstrates that small models (1.5B) can achieve reasoning capabilities comparable to or exceeding much larger models (32B+) through efficient training strategies rather than just scaling parameters.
Solving the "Hard Problem" Bottleneck: It offers a practical solution to the sparse reward problem in RL, enabling models to learn from tasks they previously could not solve at all.
Paradigm Shift: The work challenges the notion that RL only exploits pre-existing knowledge. By using partial solutions to guide exploration, it shows RL can genuinely expand a model's reasoning capacity.
Broad Applicability: While focused on math, the authors suggest the framework is applicable to coding, formal logic, and other agentic tasks where step-by-step reasoning is required.

In summary, QUESTA bridges the gap between the need for difficult training data and the inefficiency of sparse rewards, providing a robust, theoretically grounded method to scale reasoning capabilities in LLMs.