Context Selection for Hypothesis and Statistical Evidence Extraction from Full-Text Scientific Articles

1. Problem Definition

The paper addresses the challenge of automatically extracting structured empirical arguments from full-text scientific articles. Specifically, the goal is to link a high-level finding stated in an article's abstract to two components in the paper body:

1. The Core Hypothesis ($H$): A testable expectation or interpretive statement (often paraphrased and distributed across sections like Introduction or Discussion).
2. The Statistical Evidence ($E$): Hybrid numeric-textual statements (e.g., p-values, confidence intervals, coefficients) that support or refute the hypothesis, typically found in the Results section.

Key Challenges:

• Document Length & Distribution: Scientific arguments are distributed across sections, making it difficult to locate specific hypotheses and evidence within long documents.
• Rhetorical Ambiguity: Many paragraphs are topically related to the abstract finding but serve different rhetorical roles (e.g., background vs. hypothesis vs. results), creating "hard negatives" for retrieval systems.
• Extraction Complexity: While hypothesis extraction is a semantic alignment task, evidence extraction requires parsing complex, structured numeric data, which is prone to errors even when the correct text is provided.
• Limitations of Full-Text Prompting: Despite Large Language Models (LLMs) having large context windows, feeding entire papers often leads to "signal dilution," where relevant information is lost amidst irrelevant but topically similar text.

2. Methodology

The authors propose a sequential two-stage retrieve-and-extract pipeline within a Retrieval-Augmented Generation (RAG) framework.

The Pipeline

1. Stage 1 (Hypothesis Extraction):
   • Query: The abstract finding ($F_A$).
   • Task: Retrieve candidate paragraphs from the full text and extract the core hypothesis ($\hat{H}$).
2. Stage 2 (Evidence Extraction):
   • Query: A composite query formed by concatenating the abstract finding and the extracted hypothesis ($q_2 = [F_A; \hat{H}]$).
   • Task: Retrieve candidate paragraphs and extract the statistical evidence ($\hat{E}$).

Context Configurations (Variables)

The study systematically varies how context is constructed to isolate the effects of retrieval quality and quantity:

• Full-Text Baselines: Zero-shot and Chain-of-Thought (CoT) prompting using the entire article body (excluding abstract/bibliography).
• Standard RAG: Retrieving top-$k$ paragraphs ($k \in \{5, 10, 20\}$) using a pre-trained SciBERT retriever.
• RAG + Reranker: Retrieving a larger pool ($M=30$) with SciBERT, then using an LLM (RankGPT) to rerank and select the top-$k$ ($k=5$).
• Fine-tuned Retriever + Reranker: A task-specific retriever fine-tuned on a multi-task triplet loss to distinguish rhetorical roles (hypothesis vs. evidence) using hard negatives (semantically similar but rhetorically different paragraphs).
• Oracle Setting: Providing the extractor with the gold (human-annotated) paragraph to establish an upper bound and separate retrieval failures from extraction limitations.

Evaluation Metrics

• Hypothesis Extraction: Uses a calibrated semantic matching criterion. Predicted sentences are compared to gold sentences using embeddings (Gemini) with a threshold ($\tau=0.89$) to account for paraphrasing.
• Evidence Extraction: Uses component-level scoring. An LLM parser extracts structured fields (variables, effect relationships, confidence intervals, test statistics, sample size). Precision/Recall/F1 are calculated based on the overlap of these specific components, allowing for partial credit.

3. Key Contributions

1. Task Decomposition: The paper formalizes the extraction of "claim traces" as a sequential two-stage problem, distinguishing between the semantic nature of hypothesis extraction and the structured nature of evidence extraction.
2. Controlled Ablation Study: It provides a rigorous comparison of context selection strategies (quantity, quality, and oracle settings) across four different LLM extractors, isolating the specific bottlenecks of retrieval vs. extraction.
3. Fine-Tuned Retriever: Development of a specialized retriever trained to distinguish between paragraphs that are topically similar but rhetorically distinct (e.g., distinguishing a background summary from a specific hypothesis).
4. Diagnostic Framework: Introduction of metrics like "Relevant Sentence Proportion" (RSP) and transition analysis to understand how retrieval changes impact performance.

4. Key Results

The experiments were conducted on 2,257 social and behavioral science articles.

• Hypothesis Extraction:

  • Retrieval is Critical: Targeted context selection significantly outperforms full-text prompting. The best non-oracle configuration (Fine-tuned Retriever + Reranker) achieved a mean F1 of 0.50, compared to 0.39 for full-text prompting.
  • Context Quantity: Increasing $k$ (5 $\to$ 20) generally improves performance, suggesting hypotheses are often distributed across multiple paragraphs.
  • Retrieval Quality: Reranking and fine-tuning yield consistent gains by filtering out "hard negatives" (topically relevant but rhetorically incorrect paragraphs).
  • Oracle Gap: Even with perfect paragraph selection (Oracle), performance jumps to 0.73, indicating that while retrieval is a major bottleneck, the extractor still has room for improvement.

• Statistical Evidence Extraction:

  • Extraction is the Bottleneck: Evidence extraction remains substantially harder. Even with Oracle paragraphs (perfect retrieval), performance plateaus at a moderate F1 of 0.47–0.55.
  • Retrieval Impact: Improving retrieval quality helps, but gains are smaller than for hypotheses. Full-text prompting sometimes performs competitively with small-context RAG, suggesting that broader context may occasionally help models locate sparse statistical data.
  • Limitation: The moderate Oracle performance indicates that the primary failure mode is the extractor's inability to faithfully parse hybrid numeric-textual statements, not just retrieval failure.

5. Significance and Implications

• Rethinking Full-Text Processing: The study demonstrates that simply increasing context window size (full-text prompting) is not a silver bullet; in fact, it often degrades performance due to signal dilution. Targeted retrieval is essential.
• Task-Dependent Strategies:
  • For Hypothesis Extraction, the focus should be on high-quality retrieval (reranking, fine-tuning) to filter rhetorical noise.
  • For Evidence Extraction, retrieval improvements alone are insufficient. Future work must focus on extractor fidelity for structured data (e.g., schema-constrained outputs, hybrid parsing for numbers) to handle the complexity of statistical reporting.
• System Design: The paper advocates for a coupled retrieval-extraction approach where retrieval is optimized for the specific rhetorical role required at each stage of the argument trace.

In conclusion, the paper establishes that while retrieval configuration is the dominant factor for extracting hypotheses from scientific papers, the extraction of statistical evidence is limited by the model's ability to process structured numeric data, requiring new approaches beyond standard RAG.