Technical Summary: EviBack

Problem Statement

Reinforcement learning (RL) has enabled Agentic Retrieval-Augmented Generation (RAG) systems to learn multi-turn search strategies from verifiable outcome rewards. However, a critical limitation exists in current training paradigms: when a group of sampled trajectories (rollouts) yields no correct answers (an "all-zero" group), the standard RL signal provides no comparative information. In these scenarios, the normalized advantage is zero, rendering partial progress or correct intermediate search behaviors indistinguishable from total failure. Furthermore, existing methods that rely on manual prompt design for process evaluation often fail to stably characterize dimensions such as query rationality and evidence sufficiency, or they risk allowing reference answers to override judgments about insufficient evidence.

Methodology

The paper proposes EviBack, a framework designed to supply auxiliary supervision to all-zero rollout groups while preserving the integrity of verifiable Actor rewards. The methodology consists of three core components:

1. Evidence-Constrained Teacher Backoff

EviBack introduces a "Teacher" model that acts as a fallback mechanism. It is activated only when an Actor rollout group contains no trajectories with a verifiable exact match (all-zero groups).

• Two-Stage Architecture: The Teacher separates evidence assessment from answer refinement to prevent reference answers from masking evidence insufficiency.
  • Stage A (Gold-Blind): Evaluates whether the accumulated evidence visible to the Actor is sufficient to answer the question. It outputs a status: Sufficient (S), Insufficient (I), or Ambiguous (A). This stage operates without access to the ground-truth answer.
  • Stage B (Gold-Aware): Only executes if Stage A determines the evidence is not insufficient ($s_A \neq I$). It refines the answer for alignment with the reference, but strictly preserves the "Insufficient" boundary established by Stage A.
• Reward Signal: For all-zero groups, the Teacher provides a hybrid reward based on the status and a reference-aligned F1 score. This reward is scaled down (by a factor $\lambda = 0.1$) to ensure it does not override the learning signal from groups containing verified hits.

2. E2E-APE (End-to-End Automatic Prompt Engineering)

To construct the Teacher policy, the authors propose E2E-APE, a constraint-guided method for generating judge prompts.

• Process: Unlike traditional prompt optimization that maximizes final task scores, E2E-APE starts from explicit constraints required for intermediate process evaluation (e.g., query rationality, evidence effectiveness).
• Automation: Using a GPT-5.5 controller, the pipeline automatically partitions rollout data, generates candidate prompts/workflows, evaluates them against specific metrics (evidence boundary compliance, stability, cost), and selects a gated two-stage policy.
• Outcome: This process transforms a manually authored single-prompt Teacher into a frozen, versioned two-stage policy without manual intervention after the initial launch.

3. Training Protocol

The system utilizes GRPO (Group Relative Policy Optimization).

• Actor-First Precedence: If any trajectory in a group achieves a verifiable exact match, the Teacher is bypassed, and standard Actor rewards are used.
• Selective Fallback: The Teacher is invoked only for groups where all trajectories fail. The resulting rewards are normalized within the group, and the fallback advantage is scaled to maintain a lower policy-gradient contribution than groups with verified hits.

Key Contributions

1. E2E-APE: A constraint-based, end-to-end automatic prompt engineering method for generating judge prompts. It shifts the focus from final task performance maximization to satisfying constraints for intermediate process evaluation, reducing manual design costs and improving scoring stability.
2. EviBack Framework: A hybrid reward system for search-agent RAG that combines verifiable final-answer rewards with Teacher-derived process rewards. It extends RL supervision from final outcomes to the quality of the search trajectories, specifically addressing the "all-zero" group problem.
3. Evidence-Boundary Preservation: A novel two-stage Teacher design that decouples evidence sufficiency judgment from answer refinement, ensuring that reference answers cannot override judgments of insufficient evidence.

Experimental Results

The authors evaluated EviBack across seven open-domain QA benchmarks (including NQ, HotpotQA, MuSiQue, 2WikiMultiHopQA, etc.) and three Qwen3 model scales (0.6B, 1.7B, and 4B).

• Performance Gains: EviBack consistently improved F1 scores over the strong Search-R1 baseline.
  • At the 1.7B scale, EviBack achieved a 16% relative gain over the baseline.
  • Improvements were observed in both single-hop and multi-hop macro F1 scores across all scales.
• Search Efficiency: The method reduced the mean number of searches, duplicate query rates, and forced termination rates (max-turn). For instance, at the 1.7B scale, the valid-answer rate increased from 0.7317 to 0.8611, while mean searches decreased from 1.73 to 1.59.
• Teacher Construction: The E2E-APE constructed Teacher outperformed a manually designed single-prompt dual-task Teacher, improving F1 by 0.0260 and valid-answer rate by 0.2197, while significantly reducing search overhead.
• Ablation Studies: Experiments confirmed that the two-stage evidence constraint and the specific fallback scaling factor ($\lambda=0.1$) were critical for balancing performance gains with search efficiency.

Significance and Claims

The paper claims that EviBack addresses a fundamental gap in training search agents: the lack of comparative signals in failure cases. By introducing a selective, evidence-constrained backoff mechanism, the system provides fine-grained feedback on intermediate reasoning without compromising the verifiability of the final reward.

The authors emphasize that their approach:

• Restores auxiliary supervision for groups that would otherwise provide no learning signal.
• Prevents reference leakage from distorting evidence sufficiency judgments, a common issue in process-reward models.
• Demonstrates the viability of automated prompt engineering (E2E-APE) for creating complex, constraint-satisfying evaluation policies that outperform manual designs.

The work concludes that while richer Teacher-derived signals (e.g., query quality, redundancy) remain a direction for future research, the current evidence-constrained design offers a robust and effective method for improving Agentic RAG performance. The code is promised to be made publicly available at a later stage.