BioDivergence: A Benchmark and Evaluation Framework for Hidden Contextual Contradictions in Biomedical Abstracts

Technical Summary: BioDivergence

Problem Statement
Biomedical literature frequently contains findings that appear to disagree across studies. However, many of these apparent disagreements are not genuine logical contradictions; rather, they are "context-conditioned divergences" where claims remain locally valid under different conditions (e.g., varying cohorts, geographies, assay protocols, or disease subtypes). Existing Natural Language Inference (NLI) and scientific claim-verification benchmarks (e.g., FEVER, SciFact, VitaminC) typically reduce these relationships to flat labels (entailment, contradiction, neutral) or coarse conflict categories. This approach collapses context-conditioned divergence into a single "contradiction" label, overstating disagreement in the literature and failing to evaluate a critical capability for downstream biomedical NLP systems: the ability to recover the specific contextual structure that explains why two findings diverge.

Methodology
The authors introduce BioDivergence, an evaluation framework designed to operationalize the recovery of hidden contextual contradictions. The framework is built on three core components:

1. Ontology and Taxonomy:

   • Conflict Taxonomy: A six-class taxonomy distinguishing between no conflict, direct contradiction, contextual contradiction, underspecified apparent contradiction, evidence insufficiency, and unresolved scientific controversy.
   • Divergence Ontology: A 13-axis ontology capturing the contextual dimensions driving disagreement (e.g., population, geography, time period, assay protocol, study design, disease subtype, organism strain, clinical setting, etc.).

2. Task Formulation:
   The framework treats contradiction analysis as a structured multi-output problem. For each claim pair, the model must predict four primary targets:

   • Conflict Type: Selection from the six-class taxonomy.
   • Divergence Axes: Identification of the subset of axes explaining the divergence.
   • Dominant Confounder: Selection of the single most explanatory axis.
   • Reconciliation Explanation: A natural language explanation of how the claims coexist or why the tension remains.
   • (Scoped Output): Claim-level evidence-span extraction is included in v1.0 as a release-time calibration under a "claim-verbatim" silver rule, rather than as a mature grounding benchmark.

3. Benchmark Construction (BioDivergence-Silver-v1.0):

   • Data Pipeline: The benchmark is constructed from 202,180 normalized abstracts across five biomedical domains (AMR, oncology, infectious disease, genomics, clinical trials/epidemiology), yielding 527,907 claims and 50,000 candidate pairs.
   • Annotation: Labels are generated via a schema-constrained "silver" annotation pipeline using LLMs (Qwen2.5-72B for pilot, Qwen2.5-7B for full release) with heuristic fallbacks and schema validation.
   • Split Design: A critical innovation is the article-disjoint primary split (11,865 pairs: 10,183 train / 840 dev / 842 test). This ensures zero overlap of articles, claims, or pair IDs across splits. This design prevents models from achieving high scores via article-level memorization, forcing them to learn contextual reasoning. A legacy pair-level variant is retained only for backward comparability.

Key Contributions

1. Formalization of Contextual Contradiction: The paper formalizes the task of recovering why biomedical claims diverge, addressing a gap left by flat NLI and knowledge-conflict benchmarks.
2. Structured Evaluation Framework: Introduction of a six-class conflict taxonomy and a 13-axis divergence ontology, coupled with a four-primary-plus-one-scoped task specification.
3. BioDivergence-Silver-v1.0 Release: A large-scale, article-disjoint silver benchmark with explicit leakage audits and sensitivity analyses.
4. Evaluation Lens: Demonstration that article-disjoint evaluation fundamentally shifts model rankings, separating article memorization from contextual reasoning learning.

Experimental Results

• Performance Drop due to Leakage Control: When moving from the legacy pair-level variant (which allows article overlap) to the article-disjoint primary split, the fine-tuned reference model's contextual-contradiction F1 drops by approximately 12 points (from ~0.521 to ~0.401). This indicates that a significant portion of performance in previous pair-level benchmarks was driven by article-level memorization.
• Zero-Shot Performance: The full 842-example primary test evaluation with Mistral-7B-Instruct-v0.3 (zero-shot) achieves 0.5523 accuracy and 0.3894 contextual-F1. This confirms that the signal is recoverable by annotator-family-independent baselines without article leakage.
• Task Difficulty: Divergence-axis extraction and dominant-confounder prediction remain difficult for all systems, lagging behind conflict-type accuracy. This suggests that recovering why claims diverge is harder than deciding whether they do.
• Model Specialization: Zero-shot LLMs dominate coarse classification and reconciliation generation, while fine-tuned reference models perform better on contextual detection and confounder prediction on legacy splits. Retrieval-based baselines dominate evidence-span extraction (by construction of the verbatim rule).
• Reliability: The benchmark reliably supports evaluation for two conflict classes (no conflict and contextual contradiction) and nine of the thirteen divergence axes in v1.0.

Significance and Claims
The paper claims that BioDivergence provides a necessary evaluation lens that separates contextual divergence from direct contradiction. By enforcing article-disjoint splits, it isolates article-level memorization from task learning, offering a more honest assessment of a model's contextual reasoning capabilities. The authors position BioDivergence-Silver-v1.0 not as a gold-standard expert-curated resource, but as a structured, large-scale silver benchmark that exposes multi-component reasoning tasks previously unevaluated. The framework highlights that flat contradiction labels are insufficient for biomedical NLP, as they obscure the nuanced, context-dependent nature of scientific disagreement. Future work is noted to extend annotation, grounding, and gold calibration, but v1.0 is presented as a functional tool for leakage-aware evaluation.