The Alignment Bottleneck in Decomposition-Based Claim Verification

Technical Summary: The Alignment Bottleneck in Decomposition-Based Claim Verification

1. Problem Statement

Verifying complex, multi-faceted claims is a significant challenge in fact-checking because these claims often conflate multiple distinct facts. A common strategy to address this is claim decomposition, where a complex claim is broken into smaller, atomic "sub-claims" that are verified individually.

However, empirical results regarding the effectiveness of decomposition have been inconsistent. Some studies suggest it improves performance, while others claim it is unnecessary or even detrimental. The authors argue that these inconsistencies are not due to the decomposition itself, but rather to two overlooked "bottlenecks":

• Evidence Alignment: How granular the evidence is relative to the sub-claims.
• Sub-claim Error Profiles: How errors in sub-claim labeling (noise) propagate to the final claim-level verdict.

2. Methodology

The researchers developed a framework to isolate these two bottlenecks through a controlled experimental setup.

A. Dataset Creation:

• They introduced a new dataset derived from PHEME (rumor verification), featuring real-world, temporally bounded claims.
• Unlike previous datasets that use automatically extracted evidence, this dataset includes human-annotated sub-claim evidence spans and fine-grained veracity labels (True, False, Unverified).
• They also tested generalizability using MMM-Fact and COVID-Fact datasets.

B. Experimental Setups (The Two Bottlenecks):
To study evidence alignment, they defined two configurations:

1. Sub-claim Aligned Evidence (SAE): Each sub-claim is paired specifically with its relevant evidence span (Granular).
2. Repeated Claim-level Evidence (SRE): The entire set of claim-level evidence is repeated for every sub-claim (Coarse).

To study error propagation, they compared two supervision regimes:

1. Oracle Labels: Using gold-standard, human-annotated sub-claim labels (Upper bound).
2. Predicted/Noisy Labels: Using labels generated by models (GNN or LLM) to simulate real-world error propagation.

C. Models Used:

• Sub-claim Verifiers: Graph Neural Networks (GNN), BERT-based encoders (CHEF), and Large Language Models (Qwen3-14B).
• Claim Verifiers: Qwen3-14B reasoning model.

3. Key Contributions

• New Benchmark: A high-quality dataset of complex claims with manually annotated sub-claim evidence and veracity labels.
• Formalization of Alignment: A rigorous distinction between SAE and SRE setups to quantify the impact of evidence granularity.
• Error Propagation Analysis: A systematic study of how different types of sub-claim errors (e.g., aggressive refutation vs. conservative abstention) affect the final claim verdict.

4. Results

• The Alignment Requirement: Decomposition only provides significant performance gains when evidence is granular and strictly aligned (SAE). In the SRE setup (repeated claim-level evidence), decomposition often fails to improve performance and can actually degrade it (as seen in MMM-Fact and COVID-Fact).
• The Impact of Noise: When sub-claim labels are noisy (predicted rather than oracle), the SRE setup suffers a total collapse in performance. The SAE setup is more robust but still experiences degradation.
• Error Profile Sensitivity: The nature of the error matters more than the accuracy score.
  • Conservative models (like GNN) tend to predict "Unverified" (abstention). This acts as a safety mechanism that prevents incorrect polarity signals from ruining the final verdict.
  • Aggressive models (like Qwen) are more likely to commit to a "True" or "False" label. While this increases coverage, it risks propagating incorrect refutations to the claim level.

5. Significance

The paper shifts the focus of claim verification research from "how to decompose claims" to "how to align evidence and manage uncertainty."

The findings suggest that for decomposition-based systems to be viable in the real world, developers must prioritize:

1. Precise Evidence Synthesis: Moving beyond simple retrieval to finding specific evidence spans that match sub-claims.
2. Calibrated Abstention: Training sub-claim verifiers to "abstain" (label as Unverified) when uncertain, rather than forcing a binary True/False prediction, to prevent catastrophic error propagation in the final decision-making pipeline.