Context Convergence Improves Answering Inferential Questions

Technical Summary: Context Convergence Improves Answering Inferential Questions

Problem Statement
While Large Language Models (LLMs) are widely deployed in open-domain Question Answering (QA), their capability to handle inferential questions—where answers must be derived by integrating textual clues, background knowledge, and reasoning rather than retrieved explicitly—remains underexplored. Existing retrieval methods often rely on cosine similarity to select context passages, a metric that may not effectively capture the logical constraints necessary for narrowing down candidate answers in inferential tasks. The authors investigate how the structure and quality of context passages, specifically the convergence of sentences (hints), influence LLM performance on such questions.

Methodology
The study utilizes the TriviaHG dataset, which contains questions and associated hints (sentences). The authors propose a novel approach to passage construction based on the metric of convergence:

1. Convergence Metric: Adapted from prior work, convergence measures how effectively a hint eliminates incorrect candidate answers. The calculation involves:
   • Generating a set of plausible candidate answers for a question.
   • Evaluating whether each candidate is consistent with a specific hint.
   • Computing a score based on the proportion of incorrect candidates eliminated. A score of 1 indicates the hint uniquely isolates the correct answer, while 0 indicates no narrowing effect.
2. Passage Construction: The authors create passages by combining subsets of 3–5 sentences from the dataset. These subsets are categorized into High-score (high average convergence) and Low-score (low average convergence) groups. A control group is selected based on cosine similarity between the question and the sentences.
3. Experimental Setup:
   • Models: Six LLMs of varying sizes and architectures (LLaMA 3.2 1B, Gemma 3 1B, Qwen 3 4B, Gemma 3 4B, Qwen 3 8B, LLaMA 3.1 8B) were evaluated.
   • Prompting: A few-shot prompting strategy (Figure 2) was used to instruct models to provide exact answers based on the provided context, avoiding explanations or "I cannot" responses.
   • Ordering Analysis: The study further examined the impact of sentence ordering by arranging passages in descending (high to low convergence) and ascending (low to high convergence) orders.
   • Metrics: Performance was measured using ExactMatch, Precision, Recall, and F1 scores.

Key Results

• Convergence vs. Cosine Similarity: Passages constructed from sentences with higher convergence scores yielded substantially better answer accuracy across all six models compared to passages selected by cosine similarity. Specifically, the High-score convergence group consistently outperformed the Low-score group and the cosine-based groups in ExactMatch and F1 metrics. This suggests that convergence captures a form of relevance specifically tied to inferential reasoning that cosine similarity misses.
• Sentence Ordering: Ordering sentences by descending convergence (placing the most constraining hints first) resulted in slight but consistent performance gains over ascending order. This indicates that LLMs exhibit a positional bias, placing greater weight on earlier, information-rich cues during the inference process.
• Model Robustness: The positive impact of high-convergence passages was observed across models of different parameter sizes (1B to 8B) and architectures, suggesting the finding is generalizable.

Contributions
The paper makes two primary contributions:

1. Convergence as a Selection Signal: It introduces and systematically validates convergence as a novel and effective signal for selecting sentences for passage construction in inferential QA. The authors demonstrate that selecting sentences based on their ability to eliminate incorrect candidates leads to significantly higher accuracy than conventional semantic similarity methods.
2. Analysis of Sentence Ordering: It analyzes the role of sentence ordering in passage construction, showing that arranging hints in descending order of convergence yields modest performance improvements. This highlights a positional bias in LLMs, where earlier, high-convergence cues are prioritized during reasoning.

Significance and Claims
The authors claim that convergence serves as a practical and robust signal for guiding passage construction, particularly for tasks requiring reasoning beyond simple extraction. The findings suggest that for inferential questions, the "quality" of a context sentence is better defined by its ability to constrain the answer space (eliminate wrong answers) rather than its semantic similarity to the query.

The paper concludes that constructing passages from highly convergent sentences and ordering them by descending convergence is the optimal strategy identified for supporting inferential reasoning in LLMs. The authors emphasize that these findings help in analyzing how LLMs process structured contextual information and provide a basis for improving Retrieval-Augmented Generation (RAG) and Retrieval-Augmented Reasoning (RAR) systems. The work does not claim to solve all inferential QA challenges but offers a specific, measurable improvement in context selection and ordering. Future work is noted to involve replacing cosine similarity with stronger retrieval models like DPR and ColBERT to further study retrieval strategies.