Large-Scale Reproduction of Alzheimer Research on aCommon Dataset Reveals an Information Ceiling on theField’s Evidence

Technical Summary: Large-Scale Reproduction of Alzheimer Research on a Common Dataset

1. The Problem

Scientific fields often accumulate vast numbers of published claims that are difficult to synthesize or compare directly. In Alzheimer's disease (AD) research, tens of thousands of publications propose competing hypotheses (e.g., amyloid, tau, vascular, neuroinflammation) regarding disease mechanisms. However, standard replication efforts are limited by scale and methodology:

• Scalability: Manual replication cannot keep pace with fields producing hundreds of thousands of claims.
• Comparability: Meta-analyses aggregate estimates but fail to make rival hypotheses directly comparable when those estimates originate from different datasets, protocols, and analytic conditions.
• Adjudication Gap: A field may reproduce a model's predictive performance (e.g., AUC) on a common dataset, yet fail to determine if the underlying substrate contains sufficient information to distinguish between competing hypotheses. The paper argues that "reproduction" (predicting the outcome) is distinct from "adjudication" (distinguishing between rival causal claims).

2. Methodology: AI-mediated Hypothesis Aggregation (AHA)

The authors introduce AHA, a supervised, human-governed instrument designed to turn a literature into a testable account of what its evidence can and cannot distinguish. AHA operates in three phases using a strictly separated two-layer architecture:

• Layer 1: Interpretive (LLM Extraction)

  • Uses a hosted LLM (Llama-70B) confined strictly to extracting mechanism–outcome claims from review literature.
  • The LLM is not permitted to generate numerical conclusions.
  • Outputs are harmonized into "camps" (hypothesis groups) and mapped to executable signatures.

• Layer 2: Quantitative (Deterministic AI/ML)

  • Uses five base model classes (gradient boosting, random forest, multilayer perceptron, logistic regression, XGBoost) and one ensemble model.
  • Re-instantiates the extracted claims on a common substrate (ADNI) under a matched protocol.
  • Computes discrimination, uncertainty, and information-sufficiency diagnostics.

The Three Phases of Execution:

1. Phase 1 (Reproduction): Reconstructs hypothesis camps, maps them to executable signatures, and reproduces claims statically and dynamically on the common substrate to verify if reported predictive performance can be replicated at scale.
2. Phase 2 (Information Sufficiency Audit): Applies nonparametric, pre-model metrics to determine if the common substrate contains enough information to adjudicate the hypotheses. This relies on the Neyman–Pearson lemma and data-processing inequalities, which dictate that no model can extract more discrimination than the information present in the substrate itself.
3. Phase 3 (Behavioral Analysis): Compares the reproduced evidence against the field's publication and citation patterns to identify decoupling between attention and evidentiary support.

Key Metrics Introduced:

• Fait-Accompli Index (FAI): Measures if discrimination is driven by accumulated state (level) rather than live trajectory (slope). High FAI indicates the signal is observed after the transition event.
• Local Non-Identifiability (PBR): Assesses if converters and non-converters are locally separable.
• Residual Predictive Advantage (RPA): Measures model performance relative to a leakage-free local outcome-rate baseline.
• Mutual Information (I): Quantifies how much outcome entropy the substrate resolves.

3. Key Contributions and Results

The study was applied to 95,756 publications (2005–2025), isolating 10,884 data-bearing papers and recovering six hypothesis camps: Amyloid, Tau, Vascular, Neuroinflammation, Metabolic, and LATE/Mixed Pathology. The ADNI dataset was selected as the broadest mature substrate capable of carrying all six camps.

Phase 1 Results: Reproduction of Performance

• AHA instantiated 204 executable signatures and performed approximately 1.23 million model fits.
• Without Alzheimer-specific tuning, AHA reproduced the field's reported predictive performance. The best multivariate panel achieved an AUC of 0.844, within 0.016 of the reported literature median (0.860).
• However, under strict comparability (single canonical marker per camp), discrimination narrowed (median AUC 0.682–0.783).
• The "best" panel was identical across four camps, and discrimination distributions overlapped significantly (14 of 15 pairwise comparisons). The signal resolved largely onto a single brain-integrity axis (PC1 explaining 51.0% of variance).

Phase 2 Results: Information Insufficiency

• Fait-Accompli: For four computable camps (Metabolic, Amyloid, Tau, LATE), the FAI ranged from 0.878 to 0.955. This indicates that discrimination is carried by accumulated levels (post-transition states) rather than live trajectory changes.
• Low Information Content: The substrate supplied minimal mutual information about conversion:
  • Metabolic: 0.066 bits (8.3% of outcome entropy).
  • Tau: 0.095 bits (10.9%).
  • LATE: 0.106 bits (12.2%).
  • Amyloid: 0.134 bits (15.5%).
  • All camps resolved less than 16% of outcome entropy.
• Support Gate Failures: The Vascular camp failed the support gate (level AUC 0.408, below chance), and Neuroinflammation collapsed due to lack of longitudinal sampling (n=3, 0 converters).
• Conclusion: By the Neyman–Pearson bound, no model can exceed the discrimination available in the substrate. Since the substrate is information-insufficient, the "ceiling" for distinguishing these hypotheses falls to chance.

Phase 3 Results: Citation–Evidence Decoupling

• Research attention diverged sharply from evidentiary support. The Amyloid camp accounted for ~52,706 papers and ~1.49 million citations (approx. 21× the publications of the least-researched camp).
• Despite the lack of separable discrimination on the common substrate, citation accumulation continued to favor Amyloid, indicating that attention is driven by factors other than matched-protocol evidentiary superiority (e.g., mechanism beliefs, funding structures).

4. Significance and Claims

The paper explicitly states what it does and does not claim:

• What it claims:

  • Reproduction vs. Adjudication: The field's reported predictive performance can be reproduced at scale on a common substrate, but this reproduction does not constitute adjudication. The ADNI substrate is information-insufficient to tell the competing hypotheses apart.
  • Methodological Necessity: Fields require instruments that not only reproduce claims but also audit whether the evidence base supports the comparisons being drawn. AHA provides this dual capability.
  • Generalizability: The method is not limited to Alzheimer's; preliminary application to AI literature suggests it generalizes.
  • Neutrality: AHA neither vindicates nor condemns any specific Alzheimer hypothesis. It demonstrates that the current common substrate cannot rank them.

• What it does NOT claim:

  • It is not a causal adjudication of any hypothesis.
  • It does not prove that amyloid, tau, or any other camp is false.
  • It does not claim that ADNI is the definitive dataset for Alzheimer's; it is simply the broadest mature substrate allowing for commensurability.
  • It does not claim that the LLM generated numerical results; all quantitative outputs were deterministic.

Conclusion:
The study concludes that the Alzheimer's field treats hypotheses as competing when its broadest common substrate cannot distinguish them. The divergence in research attention is not explained by the evidentiary data on ADNI, suggesting that the field's dynamics are driven by sociological and historical factors rather than a clear, data-driven hierarchy of truth. The "information ceiling" revealed by AHA suggests that without a substrate containing sufficient information (e.g., better longitudinal sampling, earlier stages, or different markers), no amount of model complexity can adjudicate these competing theories.