📊 statistics

Bayesian Variable Selection for High-Dimensional Predictors with Missing Psychometric Outcomes

This paper introduces SHIM, a novel Bayesian framework that integrates hierarchical horseshoe shrinkage with a treatment of missing data to perform structured variable selection for high-dimensional predictors and multivariate outcomes, demonstrating its effectiveness through simulations and an application to Alzheimer's disease neuroimaging data.

Original authors: Zongyue Teng, Shujie Ma, Timothy J. Hohman, Angela L. Jefferson, Panpan Zhang

Published 2026-09-11

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Zongyue Teng, Shujie Ma, Timothy J. Hohman, Angela L. Jefferson, Panpan Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern study of the human mind, researchers are no longer satisfied with looking at a single clue. Instead, they gather a vast array of information at once: detailed maps of brain structure, measurements of blood flow, genetic markers, and scores from dozens of memory and thinking tests. This approach, known as multimodal research, offers a richer picture of how biology shapes behavior. However, this abundance of data creates a significant statistical puzzle. When scientists try to connect hundreds or thousands of brain measurements to a few thinking scores, the sheer number of possibilities can overwhelm standard mathematical tools, leading to confusion rather than clarity. Complicating matters further, people often miss appointments or fail to complete certain tests, leaving gaps in the data. Traditional methods struggle to fill these gaps without introducing errors, and they often fail to recognize that brain measurements are not independent facts but are organized in natural groups, such as regions within the brain or types of biological signals.

To solve this, a team of researchers has developed a new statistical framework called SHIM. Think of this framework as a highly organized filing system designed to sort through a chaotic room of mixed-up clues. The researchers built this system to handle three specific challenges at once: the hierarchical structure of brain data, the fact that different thinking tests are related to one another, and the reality that some test results are missing. In their work, they tested this new method against older, established techniques using simulated data that mimicked real-world conditions. The results showed that SHIM was far better at finding the true connections between brain regions and thinking skills while avoiding false alarms. It successfully identified which specific brain areas mattered and which did not, even when up to sixty percent of the test scores were missing. Furthermore, the method provided a way to fill in those missing scores in a statistically sound manner, allowing researchers to use the complete data for future studies without losing the nuance of the original measurements.

The researchers applied this new tool to data from the Vanderbilt Memory and Aging Project, a long-term study of older adults designed to understand how vascular health and brain aging affect cognitive decline. In this real-world test, they examined how two different types of brain imaging—measuring the volume of gray matter and measuring blood flow—related to two distinct but connected areas of thinking: episodic memory and executive function. Episodic memory involves recalling past events, while executive function covers skills like planning and switching between tasks. The analysis revealed that the new method could pinpoint specific brain regions linked to these abilities. For instance, it identified a connection between the volume of gray matter in the left parahippocampal gyrus and memory performance, and a link between the left inferior frontal gyrus and executive function. Notably, the new method found the connection to executive function that older methods missed, suggesting that looking at memory and executive function together, rather than separately, helps reveal hidden patterns in the data.

The study also tackled a common problem in medical research: missing data. In the second part of their application, the researchers looked at a specific biomarker found in cerebrospinal fluid, a substance that surrounds the brain and spinal cord. This biomarker is known to be associated with Alzheimer's disease, but it was missing for more than half of the participants in the study. Using their new framework, the researchers generated multiple plausible versions of the missing data based on the patterns they observed in the brain scans and other available information. When they used these filled-in datasets to analyze the relationship between memory and the biomarker, they found a clear, positive association: higher levels of the biomarker were linked to better memory performance. This result aligned with established biological knowledge, whereas a standard analysis that simply ignored the missing participants failed to find a statistically significant link. The new approach not only recovered the signal but did so with greater precision, demonstrating that it can effectively handle incomplete data without discarding valuable participants.

The success of this framework relies on how it treats the data. Unlike older methods that might treat every brain measurement as an isolated fact, this new approach respects the natural organization of the brain. It understands that measurements from the same brain region are related, and that measurements from different regions within the same imaging type are also related. By grouping these measurements together, the method can borrow strength from related data points to make more accurate decisions about which connections are real. It also treats the missing test scores not as lost information, but as hidden variables that can be inferred from the rest of the data. This allows the model to learn from the entire group of participants, rather than just the subset who completed every single test. The researchers confirmed through extensive computer simulations that this approach balances sensitivity with accuracy, meaning it finds the true connections without flagging random noise as significant findings.

While the results are promising, the researchers are careful to note the boundaries of their work. The method currently requires that the brain imaging and other background information be fully available for every participant, which may not always be the case in real-world studies. It also assumes that the missing data is missing for reasons unrelated to the unobserved values themselves, a condition that, if violated, could affect the accuracy of the results. Additionally, the framework is currently designed for continuous measurements, such as scores on a scale, and may need adaptation for other types of data like yes-or-no answers. Despite these limitations, the study provides a robust new tool for neuroscientists and psychologists. It offers a way to make sense of complex, messy, and incomplete datasets, turning a chaotic collection of brain scans and test scores into a coherent story about how the aging brain functions. By providing a reliable way to handle missing data and respect the structure of biological information, this framework helps researchers draw clearer conclusions about the biological roots of human cognition.

Technical Summary: Bayesian Variable Selection for High-Dimensional Predictors with Missing Psychometric Outcomes

Problem Statement
Modern psychometric and neuroimaging studies increasingly integrate multimodal data (e.g., structural/functional MRI, genomics) to characterize cognitive functioning. These studies face three concurrent statistical challenges:

High-Dimensionality and Hierarchy: Predictors often number in the hundreds or thousands ( $P \gg n$ ) and possess intrinsic hierarchical structures (e.g., modalities $\to$ anatomical regions $\to$ specific features). Existing regularization methods often ignore these multilevel relationships or accommodate only single grouping structures.
Multivariate Outcomes: Psychometric assessments routinely measure multiple correlated domains (e.g., memory, executive function). Many high-dimensional variable selection methods are designed for univariate responses and fail to exploit dependence across outcomes.
Missing Data: Psychometric outcomes are frequently partially observed due to missed visits or incomplete assessments. Existing methods for handling missing data (e.g., multiple imputation) are generally not designed for high-dimensional variable selection with hierarchically structured predictors.

Current approaches typically address these issues in isolation, lacking a unified framework that simultaneously handles multilevel predictor structures, multivariate outcome dependence, and incomplete response data.

Methodology: The SHIM Framework
The authors propose SHIM (Shrinkage for Hierarchical predictors and Incomplete Multivariate outcomes), a Bayesian framework designed to address these challenges jointly.

Model Structure: The framework employs a multivariate regression model where $Q$ outcomes are regressed on $P$ high-dimensional predictors and $L$ low-dimensional covariates. Missing outcomes are treated as latent variables within a Bayesian data augmentation strategy.
Hierarchical Horseshoe Prior: To accommodate the nested structure of predictors (Modality $\to$ $\to$ Group $\to$ $\to$ Feature), SHIM utilizes a hierarchical horseshoe prior on the regression coefficient matrix $\mathbf{B}$ $B$ . The prior specification for a coefficient $\beta_{kq}$ $β_{k q}$ (predictor $k$ $k$ , outcome $q$ $q$ ) is:
$\beta_{kq} | \tau, \gamma_{m(k)}, \delta_{m(k),g(k)}, \lambda_k, \xi_{kq} \sim \mathcal{N}(0, \tau^2 \gamma_{m(k)}^2 \delta_{m(k),g(k)}^2 \lambda_k^2 \xi_{kq}^2)$
This structure includes:
- A global shrinkage parameter ( $\tau$ ).
- Modality-level ( $\gamma$ ) and group-level ( $\delta$ ) shrinkage parameters to induce regularization at higher hierarchy levels.
- Predictor-level ( $\lambda$ ) and outcome-specific ( $\xi$ ) local shrinkage parameters.
  All shrinkage parameters follow independent half-Cauchy priors.
Posterior Computation: Inference is performed via a Gibbs sampler. Missing outcomes are imputed from their conditional multivariate normal distributions, and regression coefficients, covariance matrices, and shrinkage parameters are updated from their full conditional distributions.
Posterior-Based Multiple Imputation: Beyond parameter inference, posterior draws of missing outcomes provide a coherent multiple imputation representation for downstream analyses requiring complete data.

Theoretical Properties
The paper establishes two key theoretical characterizations of the SHIM prior:

Dependence Structure: The multiplicative hierarchy induces dependence among coefficient magnitudes. Coefficients sharing more levels of the hierarchy (e.g., same modality and group) exhibit stronger prior correlation in their magnitudes, facilitating information sharing among biologically related variables.
Marginal Density: The marginal prior density of a coefficient exhibits a "fifth-order logarithmic singularity" at zero (due to the product of five independent half-Cauchy variables) and heavy polynomial tails modified by a fourth-order logarithmic factor. This ensures strong shrinkage of noise while maintaining substantial prior support for large effects.

Simulation Results
Extensive simulations ( $n=300$ , $P \in \{200, 400\}$ , $Q=2$ ) compared SHIM against univariate counterparts (SHIM_U), other Bayesian methods (GIGG, MBSGSSS), and frequentist regularizers (Lasso, Group Lasso).

Variable Selection: SHIM consistently achieved high True Positive Rates (TPR) with near-zero False Positive Rates (FPR) and False Discovery Rates (FDR), resulting in F1 scores close to 1.0. It outperformed competitors, particularly in scenarios with high missingness (up to 60%) and outcome-specific sparsity.
Estimation: SHIM provided accurate coefficient estimates with low bias and well-calibrated uncertainty quantification (coverage probabilities near the nominal 95% level).
Sensitivity: SHIM remained robust under high outcome correlation and ultra-high dimensionality ( $P=600$ ). Performance deteriorated under Missing Not At Random (MNAR) mechanisms, consistent with the reliance on the Missing At Random (MAR) assumption.
Downstream Imputation: In a simulation involving a downstream analysis of a partially observed biomarker, SHIM-based multiple imputation yielded estimates with low bias and reasonable coverage, outperforming complete-case analysis and MICE in the presence of high-dimensional correlated predictors.

Real-World Application
The framework was applied to data from the Vanderbilt Memory and Aging Project (VMAP, $n=288$ ) to analyze associations between multimodal neuroimaging (regional gray matter volume and cerebral blood flow) and neuropsychological outcomes (episodic memory and executive function).

Neuroimaging Associations: SHIM identified specific associations between gray matter volume in the left parahippocampal gyrus and episodic memory, and between the left inferior frontal gyrus and executive function. Notably, the executive function association was detected by SHIM but missed by the univariate SHIM_U, highlighting the benefit of borrowing information across correlated outcomes.
Biomarker Analysis: SHIM was used to impute missing Cerebrospinal Fluid (CSF) $A\beta_{1-42}$ data (53.1% missing). The resulting downstream analysis revealed a significant positive association between CSF $A\beta_{1-42}$ and episodic memory, consistent with established Alzheimer's disease literature. Complete-case analysis yielded a similar point estimate but with wider confidence intervals (statistical insignificance), while MICE produced counter-intuitive results, likely due to collinearity issues with high-dimensional predictors.

Significance and Claims
The authors claim that SHIM provides a unified statistical framework that simultaneously accommodates multilevel predictor structures, multivariate outcome dependence, and incomplete data. The method balances sensitivity with false-positive control and offers a principled approach to generating posterior-based multiple imputations for downstream analyses. The paper notes that the associated R package, shim, is publicly available.

Limitations
The authors acknowledge several limitations:

Predictors and covariates must be fully observed.
Inference relies on the MAR assumption; performance degrades under MNAR.
The current formulation assumes continuous outcomes (Gaussian); extensions to binary or ordinal outcomes are needed.
The predictor hierarchy must be specified a priori, and performance depends on the accuracy of this specification.
High-dimensional posterior theory for the joint regression and covariance parameters remains an open area for future research.

Technical Summary: Bayesian Variable Selection for High-Dimensional Predictors with Missing Psychometric Outcomes

More like this