Diagnosing Structural Failures in LLM-Based Evidence Extraction for Meta-Analysis

Technical Summary: Diagnosing Structural Failures in LLM-Based Evidence Extraction for Meta-Analysis

1. Problem Statement

Meta-analyses and systematic reviews require converting unstructured narrative scientific text into highly structured, numerically grounded study records. This process is not merely about entity recognition (identifying a variable or a number) but about relational binding (correctly linking a specific effect size to a specific independent variable, a specific statistical method, and a specific study population).

Current Large Language Models (LLMs) show promise in isolated information extraction, but it is unclear if they can maintain the structural fidelity required for meta-analysis. The core problem addressed is that LLMs often fail to preserve the complex, multi-entity relationships (tuples) necessary for downstream statistical aggregation, leading to unreliable corpus-level data.

2. Methodology

The authors propose a structural and diagnostic framework that moves away from evaluating LLMs as "end-to-end pipelines" and instead treats extraction as a progression of increasing complexity.

• Semantic Study Schema: The researchers developed a generalized, cross-domain schema that captures essential empirical elements: Study Population ($P$), Geolocation ($G$), Sample Size ($N$), Statistical Method ($A$), Variables ($IV, DV$), Scale/Unit ($S, U$), Conditions ($C$), and Effect Size ($E$).
• Tiered Query Framework: Queries are organized into three levels of increasing "arity" (number of bound elements):
  • L1 (Single-atom): Extracting isolated properties (e.g., just the sample size).
  • L2 (Multi-atom/Relational): Binding multiple properties into tuples (e.g., $IV + DV + \text{Effect Size}$).
  • C (Derived Statistical): Performing corpus-level aggregation (e.g., calculating the mean sample size) based on extracted data.
• Experimental Design:
  • Corpora: Five diverse scientific domains (Civil Engineering, Medical/Health, Agricultural, Earth/Environmental, and Social Sciences).
  • Models: Two state-of-the-art models: GPT-5.2 (closed-source) and Qwen3-VL (open-source vision-language model).
  • Input Regimes: Per-Paper (decomposed extraction per document) vs. Global (concatenating all documents into one long context).
• Evaluation: Uses tuple-level Precision, Recall, and F1 scores, requiring both semantic correctness and accurate document attribution.

3. Key Contributions

1. A Diagnostic Perspective: A shift in evaluation philosophy from "is the answer correct?" to "at what level of relational complexity does the model break?"
2. A Tiered Query Suite: A standardized set of object-centric and method-centric queries that operationalize structural complexity.
3. A Multi-Domain Benchmark: A manually curated, gold-standard dataset spanning five distinct scientific fields.
4. Empirical Error Taxonomy: Identification of specific failure modes (role reversal, binding drift, etc.) that go beyond simple recognition errors.

4. Key Results

• The "Complexity Collapse": Performance is moderate for L1 (single-property) queries but degrades sharply as the number of bound variables increases. For the highest-arity tasks (binding variables, methods, conditions, and effect sizes), reliability dropped to near-zero.
• Input Regime Impact: The Global (long-context) regime performed significantly worse than the Per-Paper regime. Long-context inputs exacerbated failures in binding and attribution.
• Error Amplification: Minor errors in upstream extraction (L1/L2) led to massive failures in downstream statistical queries (C). For example, a task with 66% extraction F1 resulted in only 20% accuracy for a simple counting task.
• Identified Failure Modes:
  • Role Confusion: Swapping Independent ($IV$) and Dependent ($DV$) variables.
  • Binding Drift: Associating a variable from one analysis with a statistical result from a different analysis within the same paper.
  • Instance Compression: Failing to extract all results in "dense" sections of a paper where many associations are reported.

5. Significance

The study concludes that current LLMs lack the structural fidelity and numerical grounding required for automated meta-analysis. The findings suggest that the bottleneck for scientific AI is not "understanding" scientific language, but the ability to maintain stable relational bindings across many-to-many relationships.

This work provides a roadmap for future research, suggesting that instead of larger models, the field needs schema-aware, neural-symbolic approaches that treat relational binding and analytical scope as primary objectives to ensure the precision and reproducibility required for evidence synthesis.