Context Shapes LLMs Retrieval-Augmented Fact-Checking Effectiveness

1. Problem Statement

The proliferation of misinformation necessitates automated fact-checking systems. While Large Language Models (LLMs) show promise in this domain, their performance is inconsistent when processing extended contexts. Prior research has identified a "lost in the middle" phenomenon in question answering, where models struggle to utilize information located in the middle of a long prompt. However, it remains unclear how these contextual limitations specifically impact fact verification, a task that often requires synthesizing and reconciling conflicting evidence from multiple sources.

The paper addresses three core research questions (RQs):

• RQ1: To what extent can LLMs verify claims using only their internal (parametric) knowledge without external evidence?
• RQ2: How does increasing context length affect fact-checking accuracy?
• RQ3: How does the relative position of relevant evidence within a long prompt influence verification accuracy?

2. Methodology

The authors conducted a controlled experimental study using a systematic approach to isolate the effects of context length and evidence placement.

• Datasets: Three standard claim verification datasets were used: HOVER, FEVEROUS, and ClimateFEVER. To ensure comparability and minimize memorization bias, the authors unified labels into a binary format (SUPPORTS/REFUTES) and subsampled FEVEROUS to match HOVER's size.
• Models: Five open-source instruction-tuned LLMs were evaluated across different parameter scales and families:
  • Llama-3.1: 8B and 70B parameters.
  • Qwen2.5: 7B parameters.
  • Qwen3: 8B and 32B parameters.
• Experimental Design:
  1. Parametric Baseline: Models were tested on claims without any retrieved evidence to establish a baseline of internal knowledge.
  2. Evidence Integration: Gold-standard evidence sentences were concatenated into a single block and provided to the model.
  3. Context Manipulation: To test RQ2 and RQ3, the authors created four context lengths (2k, 4k, 8k, and 16k tokens).
  4. Positional Control: The evidence block was inserted at 11 evenly spaced positions (0% to 100% of the context). Unrelated "filler" text (dummy Wikipedia passages) was used to maintain constant input length while varying the evidence's relative depth.
  5. Total Configurations: This resulted in 44 distinct configurations per model (4 lengths × 11 positions).

3. Key Contributions

• Comprehensive Benchmarking: The study provides a large-scale evaluation of fact-checking performance across multiple model families, sizes, and datasets, specifically focusing on the interaction between context length and evidence position.
• Quantification of "Lost in the Middle": The paper confirms that the "lost in the middle" effect is prevalent in fact-checking, not just question answering, and quantifies the severity of accuracy degradation based on evidence depth.
• Model-Specific Robustness Analysis: It highlights significant architectural differences, showing that newer models (specifically Qwen3) exhibit greater resilience to positional bias and context length compared to Llama-3.1.
• Parametric Knowledge Baseline: The work establishes that LLMs possess non-trivial factual knowledge, capable of resolving a significant portion of claims without external retrieval, though this varies by dataset difficulty.

4. Key Results

A. Parametric Knowledge (RQ1)

• Models perform significantly above chance using only parametric knowledge.
• ClimateFEVER yielded the highest parametric scores, followed by FEVEROUS, while HOVER was the most challenging.
• Qwen3-32B achieved the strongest accuracy across nearly all datasets in both claim-only and claim+evidence conditions.
• Introducing gold evidence provided systematic improvements, with the largest gains observed in FEVEROUS (up to +20% accuracy).

B. Context Length Effects (RQ2)

• General Degradation: As context length increased (from 2k to 16k tokens), verification accuracy generally declined. Longer inputs introduced interference that reduced the effective utilization of evidence.
• Non-Monotonic Behavior: The Qwen3 models (8B and 32B) exhibited a unique non-monotonic pattern, showing a local minimum at 4k tokens before partially recovering at 8k. This suggests that context degradation is not strictly linear and may interact with internal attention mechanisms.
• Stability: Qwen3-32B demonstrated the most stable performance curve across all context lengths, showing the smallest variation between best and worst-case scenarios.

C. Evidence Placement Effects (RQ3)

• Positional Bias: Accuracy was consistently highest when evidence appeared at the beginning (0%) or end (100%) of the prompt. Performance dropped significantly when evidence was placed in the mid-context.
• Model Sensitivity:
  • Llama-3.1-8B showed high sensitivity to position, with significant accuracy fluctuations.
  • Llama-3.1-70B was more robust than its 8B counterpart but still exhibited mid-context degradation.
  • Qwen3-32B displayed the smoothest accuracy curves, remaining consistently above the "Claim Only" baseline regardless of evidence position. This indicates that for this model, the presence of evidence improves performance even if it is not optimally placed.
• Dataset Sensitivity: ClimateFEVER was the most sensitive to positional and contextual variations, showing the largest gaps between best and worst configurations.

5. Significance and Implications

• System Design: The findings underscore that simply increasing context length or retrieving more evidence does not guarantee better fact-checking. The ordering of evidence is a critical design parameter. Retrieval-augmented systems should prioritize placing high-confidence evidence at the beginning or end of the prompt to mitigate "lost in the middle" effects.
• Model Selection: The study suggests that model architecture and training strategies (e.g., those in Qwen3) play a more significant role in mitigating positional bias than parameter scale alone.
• Future Directions: The authors call for research into the mechanisms behind positional robustness (e.g., attention patterns) and the extension of these findings to realistic settings with noisy or partially relevant retrieved evidence.

In conclusion, the paper demonstrates that context is not a neutral container for information in LLMs; the structural organization of the prompt fundamentally shapes the model's ability to verify facts. Effective retrieval-augmented fact-checking requires careful prompt engineering to align with the specific attention biases of the chosen model.