Technical Summary: VeriEvol

Problem Statement

Scaling reinforcement learning (RL) for visual mathematical reasoning faces a critical bottleneck: as data volume increases, the reliability of reward labels often degrades. Existing approaches typically scale supervision by trusting automated labellers or assume underlying answers are correct, failing to address the compounding effect of label noise in RL. In visual reasoning, where models must ground variables in diagrams and execute multi-step inference, unverified labels can systematically reinforce incorrect policies. Furthermore, simply generating harder questions often results in prompts that are merely longer or solvable via text priors, lacking genuine image-grounded difficulty. The core challenge is to construct a data pipeline that simultaneously scales prompt difficulty and ensures answer reliability without coupling these processes to the policy optimizer.

Methodology: VeriEvol Framework

VeriEvol is an iterative framework that decouples prompt difficulty and answer reliability before any policy update. It transforms low-difficulty image-question seeds into verified, high-quality training samples through two extensible components:

1. Type-Aware Prompt Evolution

Instead of applying a uniform evolution template, VeriEvol routes seeds into a structured tag tuple $z = (t, y, u, c)$ representing problem family, answer type, recommended tools, and visual evidence sketches.

• Route-Specific Operators: For each route, specific evolution operators rewrite the seed question $q$ into a harder, image-grounded prompt $q'$. These operators are constrained to amplify specific reasoning skills, maintain necessary visual evidence, and avoid text-only solvability.
• Question-Level Gate: Generated prompts pass through an independent gate $\phi_q$ that checks for image grounding, visual consistency, non-trivial difficulty lift, and deduplication. This ensures the image remains necessary for the solution.

2. HTV-Agent (Hypothesis–Test Verifier)

This component treats candidate answers as falsifiable hypotheses rather than ground truth. It employs a "refutation-seeking" strategy across four stages:

• Hypothesis Generation: Three independent solver branches (with varying temperatures) generate candidate answers and reasoning traces.
• Refutation-Seeking Verification: Verifiers actively search for counter-evidence using:
  • Programmatic Channels: Translating constraints into Python assertions executed in a restricted interpreter.
  • Visual Channels: Using OCR, local crops, and pixel-level probes to verify that cited quantities and symbols are grounded in the image.
• Conflict Resolution: A decider evaluates the solver hypotheses against verifier reports. It assesses the logical soundness of objections rather than blindly trusting the verifier. If a verifier refutes a solver with sufficient evidence, the answer is revised; if evidence is inconsistent, the sample is rejected.
• Deterministic Acceptance Gate: A final gate $g(\hat{a}, m)$ accepts an answer only if it passes all conjunctive checks: schema compliance, visual support, verifier approval, executable assertions, and solver-cluster consensus.

3. Data Materialization

• VeriEvol-SFT: Stores the full reasoning trace and evidence for supervised fine-tuning.
• VeriEvol-RL: Stores the prompt, verified answer, and verification metadata. The reasoning trace is removed to force the policy to generate its own during rollouts. The metadata includes a deterministic reward checker (e.g., exact match, symbolic equivalence) compatible with existing GRPO-style RL recipes.

Key Contributions

1. Scalable Multimodal Evol-Instruct Framework: A pipeline that transforms low-difficulty seeds into harder, image-grounded prompts. Experiments show that scaling SFT data from 10K to 250K samples increases mean accuracy by +19.31. At the RL stage, scaling verified data from 10K to 130K adds a smooth +4.60 gain, with the design itself contributing +3.88 over an un-evolved baseline (decomposed into +1.82 from evolved prompts and +2.06 from the verifier).
2. HTV-Agent: A modular, hypothesis-test agent-as-a-judge that screens answers through independent solvers, multi-source counter-evidence (programmatic and visual), and a decider. It allows for the addition of new verifier channels without re-engineering the policy optimizer.
3. Traceable Open Release: The authors release prompts, data, models, and the full verifier trace for every sample (including solver hypotheses, counter-evidence reports, tool outputs, and gate outcomes). This enables downstream work to audit and rescale construction decisions rather than merely inspecting final outputs.

Experimental Results

Evaluated on a five-benchmark visual-math suite (MathVista, MathVision, MathVerse-VO, DynaMath, We-Math) using a Qwen2.5-VL-7B backbone:

• Performance: VeriEvol achieves a mean accuracy of 59.12, leading three of the five benchmarks and the overall average against strong external 7B baselines. Specifically, it leads on MathVerse-VO (70.46 vs. 63.80 for the next best) and We-Math-Strict (63.05 vs. 53.81).
• Scaling Behavior:
  • SFT Scaling: Mean accuracy rises monotonically from 35.42 (10K) to 54.73 (250K).
  • RL Scaling: Verified RL data scales monotonically from 54.52 (10K) to 59.12 (130K), with no benchmark showing performance degradation as volume increases.
• Component Analysis:
  • Evolution: Evolved prompts alone add +1.82 to the RL baseline. Training dynamics show evolved prompts sustain higher policy entropy and achieve higher terminal rewards, preventing early convergence to low-entropy modes.
  • Verification: The HTV-Agent adds +2.06. Ablation studies on the verifier as an inference-time judge show that multi-solver self-consistency benefits visual-rich tasks, while programmatic execution benefits computation-heavy tasks, confirming the complementarity of evidence channels.
• Error Analysis: A pilot audit on 60 samples indicates a shift in error distribution. Visual grounding (VG) and text-only shortcut (TS) errors decreased significantly (from 57% to 38% of residual errors), while the system's remaining errors shifted toward reasoning chain and symbolic/arithmetic limitations inherent to the 7B backbone capacity.

Significance and Claims

The paper posits that scaling visual reasoning requires treating data construction as a verifiable problem rather than a policy-side optimization problem. By decoupling prompt evolution from answer verification, VeriEvol ensures that reward signals remain reliable as data volume grows.

• Orthogonality: The framework is orthogonal to policy-side innovations (e.g., GRPO variants, reward shaping), allowing future optimizers to build directly on VeriEvol-RL data.
• Composability: The modular nature of the verifier allows for the addition of targeted refutation channels (e.g., new tools or evidence types) to address specific failure modes without redesigning the training loop.
• Transparency: By releasing full verifier traces, the authors aim to shift the focus of downstream research from inspecting outputs to auditing the construction pipeline itself, enabling more robust scaling and verification of multimodal reasoning systems.

The authors acknowledge limitations, including dependence on seed-pool coverage, the potential for shared blind spots between solvers and verifiers, and the fact that the RL-ready subset is narrower than the SFT-ready subset due to stricter verification requirements. They emphasize that the observed gains are suggestive of trends rather than statistically significant differences at the single-benchmark level for small margins.