🤖 AI

Grounding Multi-Hop Reasoning in Structural Causal Models via Group Relative Policy Optimization

This paper proposes a novel framework that grounds multi-hop fact verification in Structural Causal Models and optimizes reasoning chain complexity through a Rule-based Group Relative Policy Optimization (GRPO) strategy, significantly outperforming state-of-the-art baselines by addressing hallucinations and the inverted U-shaped relationship between reasoning depth and accuracy.

Original authors: Yunhan Bu, Quan Zhang, Huaping Zhang, Guotong Geng, Chunxiao Gao, Askar Hamdulla, Juan Wang, Qiuchi Li, Baohua Zhang, Shuai Lei, Yunbo Cao, Zhunchen Luo

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yunhan Bu, Quan Zhang, Huaping Zhang, Guotong Geng, Chunxiao Gao, Askar Hamdulla, Juan Wang, Qiuchi Li, Baohua Zhang, Shuai Lei, Yunbo Cao, Zhunchen Luo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a complex mystery. You have a claim (the "crime") and a pile of scattered evidence (witness statements, photos, documents). Your job is to figure out if the claim is true or false.

For a long time, AI detectives (Large Language Models) have been good at reading, but they often get lost in the details. They might make up facts ("hallucinations") or jump to conclusions without connecting the dots properly. They might say, "The suspect was at the park, and the park has a red bench, so the suspect must have stolen the watch!" without ever proving the connection between the bench and the watch.

This paper introduces a new way to train AI detectives to be more logical, transparent, and accurate. Here is the breakdown of their method, using simple analogies:

1. The Problem: The "Over-Thinker" Trap

The researchers noticed something strange. When they asked AI to explain its reasoning step-by-step (like writing out a long thought process), accuracy didn't just keep going up. Instead, it followed an "Inverted U-shape."

Too short: The AI didn't think enough and missed clues.
Just right: The AI thought clearly and got the answer.
Too long: The AI started rambling, getting confused, and making up connections that didn't exist. It was like a detective who talks so much they forget the actual evidence.

2. The Solution: The "Causal Blueprint" (SCM)

To fix this, the authors gave the AI a Structural Causal Model (SCM). Think of this as a strict architectural blueprint for building a house of logic.

The Foundation (Exogenous Variables): These are the raw facts you find in the evidence. You can't invent them; you must find them in the documents.
The Walls (Endogenous Variables): These are the intermediate conclusions you draw from the foundation.
The Rules (Structural Functions): This is the most important part. The blueprint says: "You cannot build a wall (a conclusion) unless you have the specific bricks (evidence) to support it."

This forces the AI to stop guessing. It can't say "A leads to Z" unless it has proven "A leads to B" and "B leads to Z." It turns the reasoning process into a logical construction site where every step must be supported by the previous one.

3. The Training: The "Teacher" and the "Student"

Since the AI needs to learn how to use this blueprint, the researchers used a distillation process:

The Teacher: A very smart, large AI model was asked to solve the mysteries while explicitly writing out the blueprint (listing the evidence, the steps, and the rules).
The Student: A smaller AI model was then trained to copy this structured thinking. It learned to say, "Here is the evidence, here is the step I took based on that evidence, and here is my conclusion."

4. The Fine-Tuning: The "Strict Coach" (GRPO)

Even with the blueprint, the student AI sometimes got too wordy or made mistakes. To fix this, the researchers used a technique called Group Relative Policy Optimization (GRPO).

Imagine a coach training a team of runners. Instead of just telling one runner, "You did okay," the coach lines up five runners who all started the same race.

The coach compares them: "Runner A took the shortest, most direct path. Runner B ran in circles. Runner C hallucinated a shortcut."
The coach rewards the runner who was most efficient and accurate relative to the others.

This "Group" approach helps the AI learn to be concise. It learns that a short, logical path is better than a long, confusing one. The AI gets a "reward" for:

Getting the right answer.
Using the most direct evidence (not over-complicating things).
Keeping the reasoning chain at a "Goldilocks" length—not too short, not too long.

The Result

When they tested this new "SCM-GRPO" detective on famous logic puzzles (datasets called HoVer and EX-FEVER), it beat all the previous best methods.

It was better at solving 3-hop and 4-hop mysteries (where you have to connect four different pieces of information).
It made fewer mistakes and didn't "hallucinate" as much.
Most importantly, its reasoning was transparent. You could look at its "blueprint" and see exactly how it got the answer, making it a much more reliable tool for fact-checking.

In summary: The paper teaches AI to stop rambling and start building its arguments like a careful architect, using a strict set of rules and a smart coaching system to ensure every step is grounded in real evidence.

Technical Summary: Grounding Multi-Hop Reasoning in Structural Causal Models via Group Relative Policy Optimization

1. Problem Statement

Multi-Hop Fact Verification (MHFV) requires models to retrieve discrete evidence from multiple sources and synthesize coherent reasoning chains to verify complex claims. While Large Language Models (LLMs) have been integrated into this domain, they face significant limitations:

Logical Fragility: Existing methods, including Supervised Fine-Tuning (SFT) and Chain-of-Thought (CoT) approaches, often learn statistical correlations rather than rigorous logical mechanisms. This leads to fractured logical chains and "hallucinated reasoning."
Lack of Causal Modeling: Current frameworks (e.g., Graph Neural Networks, symbolic programs) improve evidence integration but fail to explicitly model the intrinsic causal dependencies between evidence and claims, resulting in opaque verification processes.
The Complexity Paradox: The authors empirically identify an "inverted U-shaped" correlation between reasoning chain length and accuracy. While longer chains initially improve performance, excessive structural complexity introduces noise and degrades accuracy, particularly in complex scenarios (e.g., 3-hop or 4-hop tasks).

2. Methodology

The proposed framework, SCM-GRPO, addresses these issues through a two-stage training pipeline that combines Structural Causal Models (SCM) with Group Relative Policy Optimization (GRPO).

2.1. Structural Causal Model (SCM) Framework

The authors formalize MHFV as a constructive causal inference process rather than a simple label prediction task. The reasoning process is modeled as a Directed Acyclic Graph (DAG) defined by the tuple $\mathcal{M} = (U, V, F)$ :

Exogenous Variables ( $U$ ): Objective facts retrieved from evidence documents (root nodes).
Endogenous Variables ( $V$ ): Intermediate conclusions and the final verdict derived via logical deduction.
Structural Functions ( $F$ ): Mappings that explicitly define the causal dependency of endogenous variables on their parents.

A critical constraint is enforced: a new endogenous variable can only be added to the graph if its requisite parent set is fully present. This ensures topological order, eliminating logical leaps and implicit assumptions.

2.2. Data Construction via Distillation

To train models on this structured paradigm, the authors employ a distillation pipeline:

Seed Data: Utilizes high-quality multi-hop benchmarks (HoVer, EX-FEVER).
Teacher Distillation: A powerful "Teacher" LLM (e.g., Qwen3-30B-A3B) is prompted to explicitly generate SCM components ( $U, V, F$ ) rather than just a final label.
Validation: Generated chains are filtered for consistency with ground truth labels and assembled into sequential decision-making formats ( $C = \{(\tau_t, \alpha_t, o_t)\}_{t=1}^N$ ).

2.3. Supervised Fine-Tuning (SFT)

The base model is first aligned with the SCM reasoning paradigm using the constructed structured dataset ( $D_{struct}$ ). The objective is to maximize the likelihood of generating the correct structured sequence (Evidence $\to$ Derivation $\to$ Conclusion), establishing a foundation for traceable verification.

2.4. Optimization via Group Relative Policy Optimization (GRPO)

To address the trade-off between structural depth and conciseness (the "inverted U" problem), the authors introduce a rule-based Reinforcement Learning (RL) strategy using GRPO.

Group Sampling: For a given prompt, the policy generates a group of $K$ reasoning sequences.
Composite Reward Function ( $R$ ): The quality of a sequence is evaluated by three components:
1. Correctness ( $R_c$ ): Binary reward based on answer accuracy.
2. Structure ( $R_s$ ): Encourages parsimony (Occam's Razor) by rewarding a higher ratio of evidence variables ( $U$ ) to inference steps ( $V$ ), penalizing excessive "mental gymnastics."
3. Length ( $R_l$ ): Guides chain length toward an optimal interval $[l_{min}, l_{max}]$ to avoid the degradation associated with overly long chains.
Relative Advantage: GRPO updates the policy based on the relative performance of samples within the group, reducing training variance and stabilizing the learning of complex reasoning strategies.

3. Key Contributions

Formal Causal Framework: The introduction of SCM to MHFV, providing an interpretable semantic foundation that explicitly models causal dependencies between evidence and claims via a DAG.
Empirical Insight & RL Optimization: The discovery of the inverted U-shaped relationship between reasoning chain length and accuracy. The authors propose a novel GRPO-based strategy to adaptively constrain structural complexity, balancing rigor with parsimony.
State-of-the-Art Performance: The SCM-GRPO framework achieves superior performance on public benchmarks, demonstrating robust generalization and logical consistency in complex multi-hop scenarios.

4. Experimental Results

The framework was evaluated on HoVer and EX-FEVER benchmarks against diverse baselines, including Direct Prediction, ProgramFC, FOLK, RAG, and Search-o1.

Overall Performance: SCM-GRPO achieved State-of-the-Art (SOTA) results across all evaluation subsets.
- HoVer 2-hop: 73.42% (vs. 69.00% for Search-o1).
- HoVer 4-hop: 60.88% (vs. 56.60% for Search-o1).
- EX-FEVER 3-hop: 75.00% (vs. 72.80% for Search-o1).
Validation of RL Necessity: Experiments showed that applying SCM-CoT without RL optimization often degraded performance (e.g., Qwen3-14B accuracy dropped from 57.48% to 47.22% on HoVer 2-hop when using SCM-CoT alone). The full SCM-GRPO pipeline was required to convert structural potential into performance gains.
Ablation Studies: Comparing optimization strategies on the Qwen3-8B backbone, GRPO (75.42% accuracy) significantly outperformed both SFT alone (73.33%) and Context Preference Optimization (CPO) (71.91%), confirming the efficacy of group-wise relative objectives in stabilizing complex reasoning.

5. Significance and Limitations

Significance:
The paper claims to offer a reliable and interpretable solution for complex fact verification by grounding reasoning in explicit causal structures. It addresses the critical gap of "hallucinated reasoning" in LLMs by enforcing topological validity and using RL to dynamically optimize the trade-off between reasoning depth and conciseness. The work provides novel empirical insights into how structural complexity impacts LLM reasoning reliability.

Limitations:
The authors modestly acknowledge two primary limitations:

Data Dependence: The framework's performance partially depends on the quality of the automatically constructed structured training data generated via distillation.
Generalization to Ultra-Long Chains: The generalization capability for reasoning chains exceeding 4 hops remains to be further verified.

Future work is proposed to explore more robust data construction methodologies and extend the causal reasoning framework to a broader spectrum of complex reasoning tasks.