🤖 AI

From Verdict to Process: Agentic Reinforcement Learning for Multi-Stage Fact Verification

This paper introduces ProFact, an agentic reinforcement learning framework that optimizes multi-stage fact verification end-to-end by training a unified policy with process-aware rewards to overcome the limitations of isolated stage optimization and fixed heuristics.

Original authors: Rongxin Yang, Shenghong He, Siyuan Zhu, Chao Yu

Published 2026-06-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rongxin Yang, Shenghong He, Siyuan Zhu, Chao Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery. You have a claim (a statement someone made) and you need to figure out if it's True, False, or if there simply isn't Enough Evidence to decide.

In the past, AI detectives tried to solve this by following a rigid, pre-written checklist. They would break the claim into questions, look for answers, and then guess the verdict. The problem? Each step was done in isolation. The person writing the questions didn't talk to the person looking for the answers, and the person guessing the verdict didn't know how hard the others worked. It was like a relay race where the runners never passed the baton properly—they just ran their own legs and hoped for the best.

ProFact is a new way of training an AI detective using a method called "Agentic Reinforcement Learning." Here is how it works, using simple analogies:

1. The "One Brain" Approach (Unified Policy)

Instead of having separate experts for asking questions, finding evidence, and making a final judgment, ProFact trains one single AI brain to do the whole job from start to finish.

The Old Way: Imagine a factory assembly line. Worker A makes a part, Worker B paints it, and Worker C boxes it. If the box is ugly, Worker C gets blamed, but Worker A and B don't know they made a mistake.
The ProFact Way: Imagine a single master chef cooking a whole meal. If the soup is too salty, the chef knows immediately that they added too much salt. They learn to adjust the chopping, the seasoning, and the cooking all at once because they are responsible for the entire dish.

2. The "Coach" with Instant Feedback (Process-Aware Rewards)

The biggest challenge in training these AI detectives is that the "answer key" (the final truth) only comes at the very end. If the AI gets the final verdict wrong, it doesn't know why. Did it ask the wrong questions? Did it find the wrong evidence? Or did it just guess wrong at the end?

The Sparse Signal Problem: It's like playing a video game where you only get a "Game Over" screen at the very end. You don't know if you lost because you jumped too early, missed a power-up, or ran into a wall.
ProFact's Solution: The researchers gave the AI a coach who whispers feedback at every single step.
- Step 1 (Asking Questions): The coach says, "Good job breaking that big claim into small, clear questions!" (Reward for Question Quality).
- Step 2 (Finding Evidence): The coach says, "Great! You found the exact document that proves your point." (Reward for Evidence Grounding).
- Step 3 (The Verdict): The coach says, "Correct! You guessed the truth." (Reward for Final Accuracy).

This turns a vague "Game Over" into a detailed scorecard, helping the AI learn exactly which moves led to success and which led to failure.

3. Learning by Trial and Error (Reinforcement Learning)

To get really good, the AI doesn't just read a book; it plays the game thousands of times.

It tries different ways to break down a claim.
It tries different ways to search for evidence.
It compares its different attempts (like a group of students taking the same test). If one student gets a better score, the AI learns to copy that student's strategy.

The Results: Faster and Smarter

When the researchers tested ProFact, they found two major wins:

Better Accuracy: The AI became much better at figuring out the truth because it learned to coordinate its steps perfectly. It didn't just guess; it built a solid case.
Faster and Cheaper: Surprisingly, ProFact was also faster and used less computing power than the older methods. Because it learned a more efficient strategy, it didn't waste time asking unnecessary questions or reading irrelevant documents. It learned to be a lean, efficient detective.

Summary

In short, ProFact takes the chaotic process of fact-checking and turns it into a smooth, coordinated dance. By giving the AI a "coach" that praises or corrects it at every single step (not just the end), the AI learns to ask better questions, find better evidence, and make more accurate decisions, all while doing it faster than before.

Technical Summary: ProFact – Agentic Reinforcement Learning for Multi-Stage Fact Verification

1. Problem Statement

Automated fact verification in open-domain settings is inherently a multi-stage decision process involving claim decomposition, evidence gathering, and verdict prediction. While recent approaches combine Large Language Models (LLMs) with retrieval-augmented reasoning, existing methods typically optimize these stages in isolation or rely on fixed heuristics for coordination. This separation leads to suboptimal outcomes where intermediate decisions (e.g., generating verification questions) are not aligned with the ultimate goal of accurate verdict prediction. Furthermore, optimizing the entire verification trajectory is challenging due to sparse and delayed supervision: the final veracity label provides a reward only at the end of the process, making it difficult to assign credit to specific intermediate actions (such as a specific search query or decomposition step).

2. Methodology: ProFact Framework

The authors propose ProFact, an agentic reinforcement learning (RL) framework designed for end-to-end optimization of multi-stage fact verification trajectories.

2.1 Formulation as a Markov Decision Process (MDP)

ProFact models fact verification as a finite-horizon MDP ( $M = (S, A, P, r, \rho_0, T)$ ):

State ( $S$ ): Includes the claim, generated questions, retrieved evidence, QA-pairs, and the current reasoning stage.
Action ( $A$ ): Unified actions generated by a single policy $\pi_\theta$ , including generating verification questions, calling search tools, synthesizing evidence-grounded answers, and predicting the final verdict.
Trajectory: A sequence of states and actions $(s_0, a_0, r_0, \dots, s_T, a_T, r_T)$ representing the full verification workflow.

2.2 Three-Stage Verification Rollout

The agent executes a three-stage workflow mirroring human fact-checking:

Question Stage: The agent decomposes the claim into a set of verification questions ( $Q$ ).
Search Stage: For each question, the agent performs iterative retrieval from a pre-built evidence store (using semantic search via Qwen3-Embedding-0.6B and kNN) and generates evidence-grounded answers ( $A$ ). The context is reset between questions to prevent unnecessary expansion.
Verdict Stage: Based on the accumulated evidence and answers, the agent predicts the final veracity label (Supported, Refuted, Not Enough Evidence, or Conflicting Evidence).

2.3 Process-Aware Reward Function

To address the sparsity of the final label, ProFact introduces a process-aware reward function that provides dense, stage-level supervision:

Question Stage Reward ( $r_q$ ): Computed via METEOR-based matching between generated questions and gold-standard questions.
Search Stage Reward ( $r_s$ ): Computed via METEOR-based matching between generated question-answer pairs and gold-standard evidence annotations.
Verdict Stage Reward ( $r_v$ ): An indicator function ($1$ if the predicted label matches the gold label, $0$ otherwise).
Total Return: The sum of rewards across the trajectory, transforming a sparse outcome signal into intermediate learning signals.

2.4 Optimization via GRPO

The framework employs Group Relative Policy Optimization (GRPO) for training. Instead of using a separate critic network, GRPO estimates the advantage function by comparing a group of $G$ sampled trajectories for the same claim. The advantage $\hat{A}_{i,g}$ is calculated by normalizing the total return of a trajectory against the mean and standard deviation of the group. This allows for stable, end-to-end optimization of the unified policy without requiring value function approximation, which is difficult given the heterogeneous nature of stage rewards and retrieval dependencies.

3. Key Contributions

Agentic Verification Framework: ProFact unifies claim decomposition, evidence retrieval, and verdict prediction into a single policy with explicit retrieval actions, formulating verification as a long-horizon agentic decision-making problem.
Process-Aware Trajectory Optimization: The introduction of a process-aware reward function provides dense intermediate feedback, enabling the model to learn which intermediate behaviors (decomposition, search, reasoning) contribute to final accuracy.
Empirical Validation: Comprehensive experiments demonstrate that ProFact improves both verification performance and inference efficiency compared to strong baselines.

4. Experimental Results

Experiments were conducted on the AVeriTeC dataset using four open-source backbone models (Qwen2.5-3B/7B, Qwen3-4B/8B).

Performance: ProFact consistently outperformed baselines (Consistency, InFact, HerO) across all metrics.
- AVeriTeC Score: ProFact achieved the highest scores (e.g., 48.00 on Qwen2.5-7B vs. 42.80 for HerO), indicating superior end-to-end verification capability.
- Accuracy: ProFact showed improved final label prediction accuracy.
- Intermediate Quality: ProFact achieved the best Q-only METEOR scores, indicating stronger claim decomposition, and competitive Q&A METEOR scores.
Efficiency: ProFact demonstrated significant gains in inference efficiency. Compared to InFact, ProFact reduced time per claim by approximately 50–70% and substantially reduced token consumption (e.g., 0.38M output tokens vs. 0.66M for InFact on Qwen2.5-7B).
Ablation Study: Removing process rewards (w/o PR) resulted in a consistent performance drop across all backbones, confirming that final-label supervision alone is insufficient for learning effective multi-stage trajectories.
Algorithm Comparison: GRPO was found to be superior to PPO, DAPO, and GiGPO for this task, likely due to its ability to provide stable trajectory-level signals without the complexity of learning a value function for heterogeneous stage rewards.

5. Significance and Claims

The paper claims that process-aware trajectory optimization is a critical mechanism for multi-stage fact verification. By treating verification as a unified decision process rather than a pipeline of isolated modules, ProFact enables earlier decisions (like decomposition) to adapt to the requirements of later stages (like verdict prediction). The results highlight that:

Intermediate supervision is essential for effective credit assignment in long-horizon tasks.
Agentic RL with process-aware rewards can simultaneously improve accuracy and reduce inference costs by learning more targeted, less redundant verification strategies.
Larger models do not necessarily yield better evidence-grounded verification, as they may rely on parametric knowledge rather than retrieved evidence; however, the ProFact framework remains effective across different model scales.

The authors conclude that their work demonstrates the effectiveness of end-to-end reinforcement learning for optimizing complex, multi-stage reasoning workflows in automated fact verification.