Apparent 3D-structural variant-effect signal is explained by variant category, not structure: a category-matched evaluation across nine disease loci

Technical Summary: "Apparent 3D-structural variant-effect signal is explained by variant category, not structure"

Problem Statement
The classification of genetic variants as pathogenic or benign remains a central challenge in clinical genomics. To address variants outside protein-coding regions, deep-learning and biophysical models have been developed to score variants based on their predicted disruption of three-dimensional (3D) chromatin or protein structures (e.g., TAD boundaries, loop domains). However, the evaluation of these methods typically relies on marginal discrimination metrics (e.g., pooled AUC) that aggregate variants across all functional categories. The authors argue that this approach is fundamentally confounded: functional category (e.g., nonsense, promoter, intronic) is a strong predictor of pathogenicity, and structural-disruption scores are highly correlated with these categories. Consequently, a high marginal AUC may reflect a model's ability to distinguish functional categories rather than its ability to detect genuine structural disruption or per-variant pathogenicity. This mirrors the "circularity" issues previously identified in missense predictors.

Methodology
The study evaluates the ARCHCODE 3D chromatin structural-disruption score (specifically SSIM and log-SSIM metrics) using a rigorous, category-matched stratified evaluation framework.

• Dataset: The analysis utilized 24,238 high-confidence ClinVar variants (excluding conflicting and uncertain classifications) across nine disease-associated loci: BRCA1, CFTR, GJB2, HBB, LDLR, MLH1, SCN5A, TERT, and TP53. The dataset comprised 11,858 pathogenic and 12,380 benign variants.
• Evaluation Metric: Instead of a marginal AUC, the authors employed a category-matched stratified concordance statistic. This metric calculates the probability that a pathogenic variant is ranked higher than a benign variant only when both variants belong to the same functional category. This isolates the signal attributable to the score itself, removing the predictive power of the variant's category.
• Positive Controls: To validate the test design and ensure that the stratified approach does not artificially suppress all signals, two positive controls were applied under identical conditions:
  1. CADD: A supervised predictor (training overlaps with ClinVar).
  2. phyloP: An unsupervised conservation score (never trained on pathogenicity labels).
• Robustness Checks: The study tested the sensitivity of results to category granularity (e.g., removing single categories, using coarse three-group schemes) and verified that the collapse of the structural score was not driven by a single stratum.

Key Results

• Marginal vs. Stratified Performance: Marginally (pooled across all categories), the ARCHCODE structural score demonstrated discrimination (AUC = 0.754 for log-SSIM), which was lower than the predictive power of variant category alone (AUC = 0.827).
• Collapse of Signal: When evaluated within categories, the structural score's discrimination collapsed to chance levels.
  • SSIM: 0.507 (95% CI 0.500–0.514).
  • log-SSIM: 0.430 (95% CI 0.419–0.440).
  • This collapse was systematic across 7 of the 8 evaluable loci. Only TP53 retained a partial signal (dropping from 0.794 to 0.664), while HBB was excluded from per-locus analysis due to a lack of benign variants after cleaning.
• Positive Control Validation: In contrast to the structural score, both positive controls retained or improved their discrimination under the stratified test:
  • CADD: 0.989 → 0.991.
  • phyloP: 0.790 → 0.894 (showing that removing category confounding actually exposed stronger per-variant conservation signals).
• Robustness: The conclusion held across various category binning schemes. Even with coarser grouping, the structural scores (0.538–0.594) remained significantly lower than the controls (0.869–0.990).

Significance and Claims
The paper concludes that for the nine loci examined, the apparent pathogenicity signal of the ARCHCODE 3D-structural score is entirely explained by the variant's functional category, not by the structural information the score purports to capture. The authors assert that:

1. Category Confounding: Marginal AUC benchmarks are insufficient for 3D-structural scores because they credit the model for category composition rather than structural insight.
2. Specific Failure: The lack of signal is specific to the structural score; it is not an artifact of an overly conservative test, as evidenced by the survival of both supervised and unsupervised positive controls.
3. Scope of Application: The study characterizes the performance of these scores in the "common application regime" of clinical genomics, which is dominated by SNVs and small indels. The authors note that the benchmark contains too few large structural variants to test the scores' performance on their intended targets (large deletions, CTCF-anchor rearrangements), but within the clinical context of small variants, the scores add no value beyond category.
4. Recommendation: The authors recommend that category-matched evaluation with a positive control be adopted as a routine guard whenever a variant-effect score claims to capture structural mechanisms.

The paper explicitly states what it does not show: it does not claim 3D genome organization is irrelevant to disease, nor does it claim structural simulation is useless for visualization or mechanistic understanding at specific loci. The claim is strictly limited to the inability of these specific scores to provide per-variant pathogenicity ranking beyond what functional category already provides.