Technical Summary: StepGap

Problem Statement

Multi-hop Question Answering (QA) requires models to retrieve, connect, and verify evidence across multiple intermediate steps. However, current evaluation and reinforcement learning (RL) frameworks in search-interleaved systems remain endpoint-oriented, relying solely on final-answer Exact Match (EM). This approach suffers from two structural defects:

1. Lack of Differentiation: EM cannot distinguish between an evidence-grounded reasoning chain and a spuriously correct one derived from parametric memory or shortcuts.
2. Lack of Localization: When a policy fails, EM provides a zero-reward signal with no indication of which step broke the chain, preventing the model from learning step-specific repair behaviors.

Existing step-level checkers either collapse all subtasks into a single Large Language Model (LLM) call or rely on a single Natural Language Inference (NLI) call. Neither approach simultaneously emits typed labels (distinguishing specific error types) and grounds them in deterministic decisions suitable for training. Furthermore, post-hoc verifiers operate only after a complete answer is generated, missing the opportunity to redirect the generator during the reasoning process.

Methodology: StepGap

The authors propose StepGap, a hybrid NLI-LLM decision tree designed to detect step-level evidence gaps and emit one of three typed labels, each mapped to a concrete repair action:

• CONTRADICTED CLAIM (CC): The step asserts a claim conflicting with retrieved evidence. Repair: Retract.
• IRRELEVANT EVIDENCE (IE): The evidence does not address the entity or relation needed by the step. Repair: Re-search.
• MISSING BRIDGE (MB): The evidence is relevant but does not yet entail the claim. Repair: Bridging search.

Architecture

StepGap operates as a five-stage hybrid decision tree for each reasoning step $s_i$ (given accumulated evidence $e_{\le i}$):

1. Stage A (LLM): Checks Entity/Relation Alignment. Determines if the step targets the correct entity/relation before retrieval. Mis-targeting triggers a CC label.
2. Stage B (LLM): Checks Abstention Validity. Verifies if a "cannot be determined" claim is grounded in evidence. Unjustified abstention triggers CC.
3. Stage C (LLM): Checks Entity Consistency and Quote Extraction. Verifies if retrieved evidence is about the correct entity and extracts a verbatim supporting quote. Mismatch triggers IE.
4. Stage D (NLI): Performs Local Entailment. Uses a deterministic NLI model (DeBERTa-v3) to check if the quote from Stage C entails the claim.
   • Entailment $\rightarrow$ no-gap.
   • Neutral $\rightarrow$ MB.
   • Contradiction $\rightarrow$ CC.
5. Stage E (NLI): Performs Cross-step Verification. For conclusion steps lacking a local quote, checks if the claim is entailed by evidence from prior inference steps. Failure triggers IE.

Process Reward Integration

The typed output $\tau_i$ is integrated into GRPO (Group Relative Policy Optimization) as a per-step process reward:

• Base Reward ($r_{base}$): Assigns scalar values based on gap type (e.g., +0.20 for no-gap, -0.10 for IE, +0.05 for detecting a CC).
• Shaping Reward ($r_{shape}$): Conditions on the repair action taken in the subsequent step. It rewards the policy for performing the correct repair (e.g., re-searching after IE, retracting after CC) and penalizes ignoring the gap or issuing near-duplicate queries.

Key Contributions

1. Typed Gap-Action Coupling: The paper argues that typed step-level labels are only useful when coupled to specific repair actions. StepGap explicitly maps CC, IE, and MB to retraction, re-search, and bridging search, respectively.
2. Hybrid Decomposability vs. LLM-Only Cancellation:
   • On a human-annotated benchmark (181 steps), StepGap achieves an sF1 of 72.0, statistically indistinguishable from an LLM-only baseline (70.1).
   • However, ablation studies reveal a qualitative difference: removing any component from StepGap lowers performance (healthy decomposition), whereas removing three of four components from the LLM-only baseline improves performance. This indicates the LLM-only system suffers from competing-error cancellation, where internal stages mask each other's errors, making it unsuitable for reliable RL signals.
3. The Q-F1 Trap: The authors expose a metric flaw where Question-level F1 (Q-F1) is mechanically inflated by checkers that flag every step. A checker that flags 95% of steps can achieve near-ceiling Q-F1 while having poor step-level precision. StepGap avoids this by balancing step-level F1 (sF1) and Balanced Accuracy (BA).
4. Process Reward Efficacy: Using StepGap as a typed GRPO reward improves the Exact Match of Qwen2.5-7B-Instruct from 32.1 to 35.4 (across three seeds) and increases answer-claim support to 71.5%.

Results

• Checker Performance: StepGap achieves sF1=72.0, Q-F1=81.2, and BA=60.6% on the 181-step human benchmark. It maintains a healthy distribution of gap types (IE ~38%, CC ~15%, MB ~6%), whereas degenerate variants (like a single strong LLM) collapse to flagging almost everything as IE (87.8%), failing to distinguish error types.
• RL Training:
  • Typed+Shape (StepGap with shaping rewards) outperforms the Search-R1 GRPO baseline by +5.6 Avg EM (29.8 $\rightarrow$ 35.4).
  • It also outperforms the Search-R1 baseline by +10.9 EM on the strictly out-of-domain MuSiQue dataset.
  • The reward signal encourages the policy to increase search counts (from 2.5 to 2.8) and perform specific repairs rather than just avoiding errors.
• Distillation: A distilled Qwen2.5-7B student model retains 87% of the teacher's F1 score without requiring API calls during inference, making the system scalable for long RL runs.

Significance and Claims

The paper claims that StepGap provides a qualitatively different process signal compared to existing checkers. While two systems may have overlapping F1 confidence intervals, only the one with monotone component contributions (StepGap) is suitable for typed RL rewards. The LLM-only approach, despite similar aggregate accuracy, relies on error cancellation that obscures the specific nature of failures, making it unreliable for training policies to perform specific repairs.

The authors position StepGap not merely as a better detector, but as a necessary component for agentic RAG training, enabling models to learn how to fix reasoning chains (retract, re-search, bridge) rather than just learning to avoid final errors. The work highlights the necessity of step-level diagnostics over question-level metrics to avoid the "Q-F1 trap" and ensure that improvements in RL training are driven by genuine reasoning capabilities rather than spurious correlations.

Limitations noted by the authors:

• The human evaluation set is small (181 steps), leading to overlapping confidence intervals with baselines; the contribution is framed around architectural properties (decomposability) rather than a raw F1 win.
• Reward magnitudes are tuned for Qwen2.5-7B and do not transfer zero-shot to Llama-3.1-8B.
• Retraction detection relies on surface pattern matching, which could theoretically be vulnerable to reward hacking, though this was not observed in 200-iteration runs.