Technical Summary: Asking the Right Questions (ARQ)

Problem Statement

While Large Language Models (LLMs) have achieved significant progress in solving complex reasoning tasks (mathematics, coding) via post-training and test-time scaffolding, current approaches primarily focus on generating long chains of thought (CoT) to solve a problem directly or verifying proposed solutions. The ability to ask relevant, insight-generating questions that decompose a hard problem into manageable intermediate steps—a core component of human intelligence—has received less attention. Existing inference-time methods (e.g., Self-Ask, Analogical Reasoners) often rely on prompting to elicit auxiliary questions, but these methods show diminishing returns when applied to modern reasoning LLMs that already produce long CoTs. Furthermore, there is a lack of systematic study on whether LLMs can generate "stepping stones" (simpler, related subproblems) that genuinely improve the success rate of solving the original target task, and whether such generation capabilities can be learned via post-training.

Methodology: The ARQ Framework

The authors propose ARQ (Asking the Right Questions), a framework comprising two distinct modules: a question generator ($\phi$) and a problem solver ($\pi$).

1. Inference Procedure

Instead of directly solving a target problem $x$ to produce a solution $y \sim \pi(x)$, ARQ introduces a multi-step inference pipeline:

1. Generation: The generator $\phi$ produces a "stepping stone" problem $z$ conditioned on $x$ ($z \sim \phi(x)$). A stepping stone is defined as a problem relevant to $x$ but easier to solve (e.g., a special case, a simplified version, or a related subproblem).
2. Solving the Stone: The solver $\pi$ generates a solution $y_z$ to the stepping stone ($y_z \sim \pi(z)$).
3. Solving the Target: The solver $\pi$ is prompted with the original problem $x$, the stepping stone $z$, and its solution $y_z$ to generate the final solution $y$ ($y \sim \pi(x; z, y_z)$).

The framework supports two strategies for multiple stones:

• Sequential: Generating stones $z_i$ conditioned on $x$ and all previous stones ($z_1, \dots, z_{i-1}$).
• Recursive: Generating a new stone $z_i$ to help solve the previous stone $z_{i-1}$, effectively building a curriculum in reverse.

2. Post-Training and Data Curation

Recognizing that off-the-shelf LLMs generate a mixture of helpful and detrimental stepping stones, the authors propose a post-training pipeline to improve the generator $\phi$:

• Scoring Function: The quality of a generated stone $z$ is evaluated by the expected reward of the solver $\pi$ when solving the target $x$ conditioned on $z$. Formally: $S(z, x) = \mathbb{E}_{y_z, y} [R(x, y)]$, estimated via Monte Carlo rollouts.
• Supervised Fine-Tuning (SFT): The generator is fine-tuned on the highest-scoring stones generated for a dataset of target problems.
• Direct Preference Optimization (DPO): A preference dataset is constructed by pairing high-scoring stones with low-scoring stones for the same target. The generator is further fine-tuned using DPO to maximize the likelihood of generating high-scoring stones over low-scoring ones.
• Synthetic Data: The entire pipeline relies on synthetic data generation using frontier models (e.g., GPT-120B) to create the training pairs, avoiding the need for expensive human annotation of stepping stones.

Key Contributions

1. Existence and Transferability of Good Stepping Stones: The paper demonstrates that "good" stepping stones exist and are transferable. High-quality stones generated by a strong model significantly improve the success rate of solvers with varying capabilities (including smaller models), indicating that the benefit is not due to overfitting to a specific solver's biases.
2. ARQ Framework: A novel inference-time scaffold that separates question generation from problem solving, allowing for the study of stepping stone generation in isolation.
3. Post-Training Recipe: A method to fine-tune LLMs to become better stepping stone generators using synthetic data and a reward signal derived from the solver's performance. This shows that the ability to ask the right questions can be learned.
4. Multi-Stone Extension: The framework is extended to generate multiple stepping stones sequentially or recursively, showing that sequential generation yields further performance gains.

Experimental Results

The authors evaluated ARQ on three challenging math benchmarks: AIME 2024, AIME 2025, and BeyondAIME.

• Inference-Time Performance: Using a strong off-the-shelf model (GPT-120B) as the generator, ARQ improved the solver's success rate by an average of 13% when selecting the best stepping stone from 20 candidates. However, the average performance across all 20 stones was lower due to the presence of detrimental stones, highlighting the need for better generation.
• Transferability: Good stepping stones selected by a reference solver (GPT-120B) improved the performance of weaker solvers (GPT-20B, Qwen3-4B) to a similar degree as stones selected by the solvers themselves.
• Post-Training Impact: Fine-tuning Qwen3-8B and Qwen2.5-32B on the synthetic stepping stone dataset resulted in significant improvements. The fine-tuned models generated stones that improved solver performance by 3.1% and 2.9% respectively on average, outperforming the baseline "Solver Only" and the pre-trained versions.
• Multi-Stone Scaling: Extending ARQ to generate 3 sequential stepping stones further improved performance. For GPT-120B, the best-case score improved by 5.2% and the average by 3.9% compared to using a single stone. Recursive generation showed fluctuating or degraded performance compared to the sequential approach.
• Comparison: ARQ outperformed other intermediate-step methods like Analogical Reasoning, Least-to-Most, and Plan-and-Solve. It slightly underperformed Self-Refine (iterative refinement of the solution) on existing LLMs, likely because Self-Refine aligns more directly with existing RLVR training pipelines, but ARQ remains complementary and can be combined with Self-Refine for further gains.

Significance and Claims

The paper claims that ARQ offers an encouraging step into an underexplored research direction: identifying what questions are worth asking and training LLMs to generate them. The authors emphasize that:

• Stepping stones are a distinct reasoning capability: They are not merely longer CoTs but involve constructing related, easier problems to build intuition and verify strategies.
• Generative capability is learnable: The ability to ask the right questions is not an inherent property of all reasoning models but can be significantly enhanced through targeted post-training on synthetic data.
• Scalability: The approach is particularly promising for extremely challenging domains (e.g., IMO-level math, scientific research) where direct solution generation is difficult, and where humans routinely use simplified prototypes to validate ideas.

The authors remain modest about the current scale of their post-training experiments (limited by compute and data sources) and suggest future work in scaling to larger datasets, other domains like coding, and exploring online RL for question generation.