Technical Summary: Verifier-Backed Hard Problem Generation for Mathematical Reasoning

Problem Statement

While Large Language Models (LLMs) have achieved expert-level performance in solving scientific and mathematical problems, they struggle to autonomously generate valid and challenging novel problems. This capability is critical for two reasons: advancing LLM training through high-difficulty data and enabling fully autonomous scientific research where models can formulate new questions.

Existing approaches face significant limitations:

1. Human-Dependent Methods: Relying on domain experts to compose or modify problems is slow, costly, and limits the difficulty ceiling to human capabilities.
2. Naive Self-Play: Methods using a "setter-solver" duality (where a setter generates problems and a solver's failure rate determines the setter's reward) suffer from reward hacking. Setters can generate invalid or underspecified problems that the solver cannot answer, artificially inflating the difficulty reward without producing valid, challenging content. This leads to a collapse in training quality.

Methodology: The VHG Framework

The paper proposes VHG (Verifier-Backed Hard Problem Generation), a three-party self-play framework designed to decouple problem validity from problem difficulty. The framework introduces an independent Verifier module to the conventional setter-solver loop.

Core Components

1. Setter ($Q$): An LLM responsible for generating problem-reference pairs $(x, y^\star)$, where $x$ is the problem and $y^\star$ is the reference solution. Unlike standard self-play, the setter must provide a solution, enabling verifiable validity checks.
2. Solver ($S$): An LLM that attempts to solve the generated problem $x$. Its empirical accuracy, $Acc_S(x, y^\star)$, serves as the signal for problem difficulty.
3. Verifier ($V$): A module that validates the correctness of the pair $(x, y^\star)$ before it is used for training or scoring.

The Reward Mechanism

The critical innovation is the verifier-gated reward function for the setter. The setter is only rewarded if the generated pair is valid. The reward function is defined as:
$$R_Q(x, y^\star) = \mathbb{1}[V(x, y^\star)=1] \cdot (1 - Acc_S(x, y^\star))$$
Where:

• $\mathbb{1}[V(x, y^\star)=1]$ is an indicator function that is 1 only if the verifier accepts the pair.
• $(1 - Acc_S(x, y^\star))$ represents the difficulty (inverse of solver accuracy).

This design ensures that a setter cannot gain high rewards by generating invalid problems (where the solver fails by default). The solver's failure only contributes to the reward after the verifier confirms the problem is well-posed and the solution is correct.

Verifier Variants

The paper instantiates two types of verifiers to handle different task constraints:

1. Hard Verifier (Symbolic): Used for indefinite integrals. It utilizes symbolic computation libraries (e.g., SymPy) to strictly verify that the derivative of the generated antiderivative matches the integrand. This provides near-100% reliability.
2. Soft Verifier (LLM-based): Used for general mathematical reasoning. It employs an LLM-as-a-judge to validate the problem's well-posedness, the solution's correctness, and the correspondence between them. To mitigate noise, this is combined with hard-coded filters (rejecting malformed outputs, missing answers, or trivial copies) before LLM evaluation.

Training Pipeline

The process follows a consistent workflow:

1. Initialize the setter via cold Supervised Fine-Tuning (SFT) on seed problem-reference pairs.
2. The setter generates candidate pairs.
3. The Verifier gates the pairs, rejecting invalid ones.
4. Only verifier-accepted pairs are scored by the solver for difficulty.
5. The setter is updated via Reinforcement Learning (RL) using the difficulty-based reward.
6. The accepted pool is used to train the solver or construct challenge datasets.

Key Contributions

1. Three-Party Self-Play Framework: The introduction of a verifier module to the setter-solver dynamic, which prevents the generation of invalid problems and mitigates reward hacking.
2. Empirical Validation of Hard and Soft Verifiers: Extensive experiments demonstrating that VHG generates novel, challenging problems that are both valid and difficult, outperforming baselines in both symbolic (hard verifier) and general (soft verifier) domains.
3. Mechanism Analysis: A comprehensive analysis showing that the verifier is critical for ensuring quality. The study reveals that without the verifier, self-play methods (like R-Zero) tend to shift toward easier or invalid problems, whereas VHG successfully shifts the distribution toward hard, valid problems.

Experimental Results

Indefinite Integral (Hard Verifier)

• Setup: Evaluated on AntiderivBench (Qualifier/Competition) and a curated Integration Stress Test.
• Performance: VHG significantly outperformed baselines, including the state-of-the-art R-Zero.
  • Pass@1 Improvements: +16.9% (Competition), +16.6% (Qualifier), and +21.4% (Stress Test) over the base model.
  • Comparison: R-Zero failed to outperform vanilla GRPO after three iterations, highlighting the necessity of hard verification.
• Challenge Generation: VHG generated problems that remained challenging even for models 8x larger (Qwen3-32B), with Pass@1 rates below 50% for the hardest subsets.

General Mathematics (Soft Verifier)

• Setup: Evaluated on MATH, GSM8K, AMC, Minerva, Olympiad, and AIME (2024–2026) benchmarks.
• Performance: VHG enhanced performance across most benchmarks, raising the overall Pass@1 accuracy from 56.8% to 69.0%.
• Ablation: VHG outperformed R-Zero on all benchmarks except GSM8K. The authors attribute the GSM8K dip to a distribution shift, as VHG focuses on generating harder problems, while GSM8K consists of primary-school-level questions.

Weak-to-Strong Generalization

A notable finding is that a setter and solver based on a smaller model (Qwen3-4B) can generate problems that challenge significantly larger models (Qwen3-8B, 14B, and 32B). This suggests a scalable path for generating high-quality training data for stronger models using weaker models.

Significance and Claims

The paper claims that hard synthetic mathematical problems should be studied through the reward oracle that selects them, not just the generator that proposes them.

• Validity as a First-Class Gate: The primary contribution is establishing validity as a prerequisite for difficulty assessment. By ensuring that solver failure is only interpreted as difficulty after validity is confirmed, VHG avoids the collapse of the training pipeline seen in naive self-play.
• Scalability: The framework demonstrates that verifier-backed generation can produce data that pushes the capability boundaries of current LLMs, offering a path toward more autonomous scientific research and continuous model improvement without relying on static human datasets.
• Modesty on Guarantees: The authors explicitly note that their exact validity guarantees apply only to the hard-verifier setting (indefinite integrals). For general math, the results are presented as evidence of a practical soft-verifier pipeline, acknowledging that LLM judges may still accept subtle errors or underspecified problems. The work emphasizes that the quality of the verifier backend ultimately determines the ceiling of hard-problem generation.