Technical Summary: SEVA – Self-Evolving Verification Agent with Process Reward

1. Problem Statement

Large Language Model (LLM) agents face a critical reliability bottleneck: hallucination. While fact attribution verifiers serve as a safety layer to determine if an agent's claims are supported by source documents, existing solutions (e.g., MiniCheck, ClearCheck) suffer from a fundamental limitation: they emit only opaque binary labels (True/False).

This opacity creates two distinct failures in high-stakes deployments:

1. Lack of Self-Correction: When a verifier flags a claim, the agent cannot determine why it is wrong (e.g., whether a number was inflated, an entity swapped, or a qualifier fabricated), preventing autonomous correction.
2. Auditability Gap: Human operators cannot meaningfully audit the decision because the reasoning behind the label is invisible.

Furthermore, training such structured verifiers via Reinforcement Learning (RL) using standard binary rewards fails. When applied to multi-component outputs (e.g., JSON with alignments, reasoning, and labels), binary rewards cause advantage collapse in Group Relative Policy Optimization (GRPO). Because most responses in a training group receive a reward of 0 (due to parsing errors or a single wrong component), the variance within the group vanishes, causing the policy gradient to disappear and training to stall.

2. Methodology

2.1. Structured Verification Schema (SEVA)

SEVA is designed to produce structured, auditable output $v = (A, C, y, \gamma, e, s)$ rather than a single label:

• Evidence Alignment ($A$): Triples mapping claim spans to source spans with status (match, mismatch, not found).
• Reasoning Chain ($C$): Step-by-step verification tracing logic against source evidence.
• Label & Confidence ($y, \gamma$): A binary attribution label paired with calibrated confidence.
• Error Diagnosis ($e, s$): If the claim is unattributable, the model identifies one of six error types (numerical exaggeration, negation flip, scope inflation, temporal shift, entity substitution, fabrication) and suggests a fix.

2.2. Process Reward Mechanism

To overcome the advantage collapse of binary rewards, the authors introduce a Process Reward function $R$ that decomposes verification quality into five independent components plus a calibration term:
$$R = \underbrace{w_f R_f + w_a R_a + w_c R_c}_{\text{Process (70\%)}} + \underbrace{w_l R_l + w_d R_d}_{\text{Outcome (30\%)}} + R_{cal}$$

• Components:
  • $R_f$: Format validity (JSON parsing).
  • $R_a$: Alignment quality (grounding).
  • $R_c$: Reasoning chain quality (step-by-step logic).
  • $R_l$: Label correctness.
  • $R_d$: Error diagnosis accuracy.
  • $R_{cal}$: Asymmetric calibration term (rewards confident correctness, penalizes overconfident errors).
• Weighting: The process components (format, alignment, reasoning) are weighted 70%, while outcome components (label, diagnosis) are weighted 30%. This forces the agent to master the behavior of verification before optimizing for the outcome.
• Theoretical Justification: The authors prove (Proposition 1 & 2) that binary rewards on multi-component generation lead to zero variance in GRPO groups (advantage collapse), whereas process rewards maintain non-zero variance by decoupling component scores, ensuring a non-vanishing gradient.

2.3. Training Pipeline

1. Supervised Fine-Tuning (SFT): A 3B model (Qwen2.5) is fine-tuned on 4,992 structured examples annotated by GPT-4o-mini.
2. GRPO with Process Reward: The SFT model undergoes 350 steps of GRPO training. The process reward restores the gradient, allowing the model to improve structural quality and accuracy simultaneously.
3. Self-Evolution Loop: A Verify→Reflect→Probe→Refine loop is implemented.
   • Reflect: Aggregates error diagnoses to identify weak categories.
   • Probe: Generates targeted adversarial data for these weak categories.
   • Refine: Fine-tunes the model on the new data.
   • This loop was iterated four times on a 7B model variant.

3. Key Results

3.1. Performance and Structural Quality

• Accuracy: SEVA-3B trained with process reward achieves 69.0 F1 on the ClearFacts benchmark, matching GPT-4o-mini (69.8 F1) and significantly outperforming the SFT baseline (64.9 F1).
• Structural Reliability: Process reward drives format compliance from 72% (SFT) to 100% and alignment quality from 0.917 to 0.997.
• Comparison: While binary verifiers like MiniCheck-7B achieve higher raw accuracy (81.2 F1), they do so with binary outputs. SEVA matches the performance of large models while providing the auditable substrate (alignments, reasoning, fixes) required for safety-critical deployment.

3.2. Training Dynamics: Implicit Curriculum

The process reward induces an implicit curriculum:

• Phase 1 (Steps 0–150): The agent masters verification behavior (format, alignment, reasoning chains).
• Phase 2 (Steps 150–350): The agent refines verification outcomes (F1 score).
  This occurs without explicit scheduling, driven by the 70/30 reward weighting.

3.3. Self-Evolution and Specialization

Iterative refinement via the self-evolution loop revealed a structural finding: specialization, not generalization.

• Each refinement round produced a benchmark-specialist rather than a strictly better generalist.
• Trade-off: Rounds 2–4 improved performance on HaluEval (+14.9 pp) but degraded performance on TruthfulQA (−12.4 pp).
• This effect persisted even with 4x more training data (Round 4), indicating it is a data-distribution-induced phenomenon rather than overfitting. The structured output was essential to expose this per-category dynamic, which aggregate accuracy would have hidden.

3.4. Ablation Studies

• Binary Reward Failure: Replacing process reward with binary reward in GRPO resulted in <65 F1 and no structural improvement, confirming that binary rewards cause training to stall due to advantage collapse.
• Reward Sensitivity: A 70/30 split (process/outcome) yielded the best balance; shifting too heavily toward outcomes degraded structural quality, while shifting too heavily toward process reduced label accuracy.

4. Significance and Claims

The paper claims three primary contributions:

1. SEVA Agent: A structured verifier that matches the accuracy of GPT-4o-mini while producing auditable, diagnostic output (alignments, reasoning, error fixes) necessary for human-in-the-loop trust.
2. Process Reward Solution: A formal resolution to the "advantage collapse" problem in RL for multi-component generation. The authors argue that for any RL task with multi-component generation, reward granularity must match output granularity.
3. Structural Insight on Self-Improvement: The discovery that iterative self-refinement loops, when driven by targeted adversarial probes, tend to produce specialists rather than generalists. This challenges the monotonic improvement assumption common in Self-Refine and STaR literature.

The authors conclude that the five-component decomposition is a "dual-use asset": it extracts multiple gradients per response (enabling learning) and exposes per-category dynamics (enabling the discovery of specialization). They suggest that future trustworthy agent pipelines should prioritize architectural responses (e.g., per-domain routing) over simply adding more training rounds to achieve generalization.