Spending Scarce Confirmatory PET Measurements: Target-Aligned Validation in A4/LEARN

Technical Summary: Spending Scarce Confirmatory PET Measurements

Problem Statement
The paper addresses the operational challenge of allocating scarce confirmatory measurements, specifically amyloid positron-emission tomography (PET), in Alzheimer's disease research and clinical workflows. As anti-amyloid therapies require biomarker confirmation and cheaper surrogates (e.g., blood-based biomarkers) expand screening, a two-stage workflow emerges: broad screening followed by targeted confirmation. The core problem is determining which subjects should receive the expensive confirmatory PET scan when the validation budget is smaller than the screened population.

A common heuristic is to validate subjects with the highest prediction uncertainty (generic uncertainty sampling). However, the paper argues this approach is flawed when the scientific objective is not merely to improve a predictor, but to support a specific, prespecified claim (e.g., an APOE4 carrier vs. non-carrier contrast, an age slope, or a cutoff-indexed positivity rate). Generic uncertainty may spend resources on subjects that are hard to predict but irrelevant to the specific scientific or regulatory claim.

Methodology
The study utilizes the A4/LEARN (Anti-Amyloid Treatment in Asymptomatic Alzheimer's Disease) archive as a retrospective "design laboratory." The author treats the observed PET outcomes as a ground truth to evaluate candidate validation strategies without actually spending the budget.

1. Data and Targets:

   • Archive: 4,460 participants with both PET (Centiloid score) and APOE4 status.
   • Primary Endpoint: Binary PET positivity defined as Centiloid $\ge$ 24 ($Y_i = 1\{C_i \ge 24\}$).
   • Targets:
     • Primary: Finite-archive difference in PET positivity between APOE4 carriers and non-carriers.
     • Secondary: Age slope of PET positivity and cutoff-indexed PET positivity across a range of Centiloid thresholds.
   • Surrogates: First-phase variables including demographics, APOE4, cognitive scores, plasma biomarkers, and MRI.

2. Validation Designs Compared:

   • Random Validation: Baseline design-based sampling.
   • Generic Uncertainty: Selecting subjects with high residual prediction uncertainty ($\sigma_i$).
   • APOE4-Balanced: Ensuring equal expected PET counts in carrier and non-carrier strata.
   • Target-Specific Validation: Allocating based on a score combining target influence ($|a_i(t)|$) and residual uncertainty ($\sigma_i(t)$). The proposed value metric is proportional to $|a_i(t)|\sigma_i(t)$.

3. Estimation Framework:
   The author employs a recorded-probability design (two-phase sampling) where validation probabilities ($\pi_i$) are recorded before PET outcomes are used. The estimator is a model-assisted Horvitz-Thompson type:
   $$\hat{\theta}(t) = \sum_i a_i(t)\mu_i(t) + \sum_i \frac{R_i}{\pi_i} a_i(t)\{Y_i(t) - \mu_i(t)\}$$
   This ensures the estimator is centered correctly regardless of surrogate model accuracy, provided the validation probabilities are known.

Key Results
The performance of different designs is evaluated at a PET budget of 200 scans, measured by the ratio of confidence interval (CI) width relative to random validation (values < 1 indicate higher efficiency).

1. Primary APOE4 Contrast:

   • Generic Uncertainty: CI-width ratio = 0.980 (minimal gain over random).
   • APOE4-Balanced: CI-width ratio = 0.923.
   • Target-Specific Scoring: CI-width ratio = 0.914.
   • Finding: For the APOE4 contrast, a simple transparent design that balances PET scans between carriers and non-carriers recovers nearly all the efficiency gains of a complex, fitted target-specific scoring model. The marginal benefit of the complex model is negligible for this specific target.

2. Heterogeneous Targets (Age Slope & Cutoffs):

   • Age Slope: Target-specific scoring yields a CI-width ratio of 0.806, significantly outperforming generic uncertainty (1.000) and simple balancing (1.001).
   • Cutoff-Indexed Positivity: Target-specific scoring achieves a ratio of 0.859 versus 0.905 for generic uncertainty.
   • Finding: When the target involves heterogeneous influence (e.g., age leverage) or is indexed across thresholds, generic uncertainty and simple balancing fail. Target-aligned scoring becomes essential.

3. Deployable Pilot Evidence:
   In a two-wave emulation (random pilot to learn residuals, followed by target-aligned validation), the target-specific approach maintained efficiency (CI-width ratio ~0.919) without requiring unobserved PET outcomes to make the second-wave decision, confirming the practical viability of the method.

Significance and Claims
The paper's primary contribution is a decision rule for scarce measurement allocation that prioritizes the scientific claim over prediction uncertainty.

• Operational Rule: Validation should be spent where it reduces uncertainty for the specific claim being tested. The "value" of validating a subject is the product of their influence on the target and their residual uncertainty.
• Practical Guidance:
  • For simple, well-defined strata (like APOE4 status), transparent balancing is sufficient and preferable due to ease of explanation and audit.
  • For complex targets (slopes, cutoff curves), target-specific residual-uncertainty scoring is necessary.
• Methodological Stance: The paper reframes the problem from "building the best predictor" to "designing the best validation." It emphasizes that while surrogates guide where to measure, they must not redefine what is being measured. The validity of the inference rests on the recorded validation probabilities, not the accuracy of the surrogate model.
• Scope: The author explicitly limits their claims to the A4/LEARN archive context and scalar/cutoff-indexed targets. They do not claim that surrogate models are unnecessary in all contexts, but rather that for specific, dominant strata, the added complexity of fitted residual scores offers diminishing returns compared to simple stratified balancing.

The paper concludes that the discipline of defining the target, identifying the anchoring protocol measurement, and recording validation probabilities before outcome use is critical for efficient and valid inference in resource-constrained biomedical studies.