Improving variable selection properties with data integration and transfer learning

1. Problem Statement

The paper addresses the challenge of variable selection (support recovery) in high-dimensional sparse linear regression, where the number of variables $p$ often exceeds the sample size $n$.

• The Core Difficulty: In high-dimensional settings, consistent recovery of the true set of active variables ($S$) requires restrictive conditions on sample size, signal strength, and sparsity. Standard methods often fail when signals are weak or sparsity is high.
• The Opportunity: Researchers frequently possess external information (e.g., results from prior studies, auxiliary datasets, or functional annotations) that suggests certain variables are more likely to be associated with the response than others.
• The Gap: While empirical success exists in using such information, there is a lack of a rigorous theoretical framework explaining how and under what conditions integrating this information improves variable selection consistency and convergence rates. Specifically, the paper focuses on Structural Transfer Learning, where variable selection results from source datasets guide selection in a target dataset.

2. Methodology

The authors propose a framework based on $\ell_0$-penalized likelihood with block-dependent penalties, motivated by Bayesian variable selection.

A. Externally-Informed $\ell_0$ Penalties

Instead of applying a uniform penalty across all variables, the method partitions variables into $b$ blocks ($B_1, \dots, B_b$) based on external information $z$.

• Model: $y = X\beta^* + \epsilon$, where $\epsilon \sim N(0, I_n)$.
• Selector: The method selects a model $M$ by maximizing:
  $$\hat{S}_{ei} = \arg\max_{M \in \mathcal{M}} \left\{ \ell(y; \beta) - \sum_{j=1}^b \kappa_j |M_j| \right\}$$
  where $\ell(y; \beta)$ is the log-likelihood, $M_j$ is the set of selected variables in block $j$, and $\kappa_j$ is a block-specific penalty.
• Bayesian Connection: The penalty $\kappa_j$ corresponds to a prior inclusion probability $\theta_j$ for variables in block $j$ via the BIC approximation: $\kappa_j \approx \frac{1}{2}\ln(n) + \ln(1/\theta_j - 1)$. This allows external information to be encoded as differential prior probabilities.

B. Oracle Analysis (Ideal Case)

The authors first analyze an "oracle" selector that knows the true sparsity ($s_j$) and signal strength ($\beta^*_{min, j}$) in each block.

• They derive sufficient conditions (Assumptions A1 and A2) for consistent recovery.
• They establish necessary conditions to show these bounds are tight.
• Key Finding: The oracle selector achieves consistency under milder conditions and at faster rates than a standard selector (which uses a single penalty $\kappa$ for all variables), provided the blocks differ in sparsity or signal strength.

C. Data-Driven Procedure (Empirical Bayes)

Since the oracle parameters are unknown, the authors propose a two-step Adaptive Informed Selector ($\hat{S}^{A,ei}$):

1. Step 1 (Estimation): Use a common initial penalty $\kappa^\circ$ to compute pseudo-posterior inclusion probabilities for all variables (via MCMC or similar methods). Estimate the number of active variables in each block, $\hat{s}_j$.
2. Step 2 (Adaptation): Set the block penalties adaptively based on the estimated sparsity:
   $$\kappa^A_j = \ln(p_j - \hat{s}_j) + \frac{1}{2}\ln(n)$$
   This effectively sets the prior inclusion probability $\theta_j \approx (p_j - \hat{s}_j + 1)^{-1}$.

D. Structural Transfer Learning (Tran-s-$\ell_0$)

A specific application where source datasets provide the external information:

• Input: Selected variable sets $\hat{S}^{(k)}$ from $K$ source datasets.
• Construction: Define Block 1 ($B_1$) as variables selected in at least one source dataset, and Block 2 ($B_2$) as the remaining variables.
• Robustness: The method is proven to be robust to negative transfer (where source and target supports do not overlap). If the source information is uninformative, the method asymptotically reverts to the performance of a standard selector, avoiding the degradation seen in some other transfer learning approaches.

3. Key Contributions

1. Theoretical Characterization of Gains: The paper provides the first rigorous theoretical proof that integrating external information relaxes the mathematical conditions required for consistent variable selection.
   • Weaker Conditions: Consistency can be achieved with smaller sample sizes or weaker signal strengths if the external information correctly identifies blocks with higher sparsity or stronger signals.
   • Faster Rates: When both informed and uninformed methods are consistent, the informed method converges to the true support at a faster exponential rate.
2. Oracle vs. Data-Driven: The authors prove that a computationally efficient Empirical Bayes procedure (learning penalties from data) achieves the same qualitative improvements as the infeasible oracle selector.
3. Robustness to Negative Transfer: Unlike many transfer learning methods for parameter estimation that require strong assumptions about design matrix similarity or identical supports, this structural transfer learning method requires no assumptions on the similarity of design matrices. It remains consistent even if the source and target supports are disjoint.
4. Computational Tractability: The approach leverages modern computational advances (MCMC, exact optimization for $p$ in the thousands) to make $\ell_0$ penalties feasible, moving beyond the limitations of $\ell_1$ (LASSO) which the authors note have inferior theoretical properties for support recovery.

4. Results

Theoretical Results

• Consistency: The probability of incorrect selection $P(\hat{S} \neq S)$ vanishes as $n \to \infty$ under conditions significantly milder than those required for standard $\ell_0$ selectors.
• Rate Comparison: The ratio of convergence rates between the informed and uninformed selectors is shown to be $o(1)$ (vanishing) in informative regimes, meaning the informed selector is exponentially faster.
• Necessary Conditions: The derived necessary conditions match the sufficient conditions up to logarithmic terms, confirming the tightness of the analysis.

Numerical Illustrations

• Simulations:
  • In scenarios where blocks are informative (different sparsity levels), the adaptive informed selector ($\hat{S}^{A,ei}$) significantly outperforms standard $\ell_0$, EBIC, LASSO, and SCAD in terms of Probability of Correct Recovery and False Discovery Rate (FDR).
  • In non-informative scenarios, the method performs comparably to the standard selector, demonstrating no penalty for using external information.
  • Transfer Learning: The Tran-s-$\ell_0$ algorithm outperforms competitors when source and target supports overlap. Crucially, in "irrelevant transfer" scenarios (no overlap), it remains robust and does not suffer from negative transfer, unlike methods that force inclusion of source variables.
• Real Data Application (Colon Cancer):
  • Task: Identify genes associated with TGFB in humans ($n=262, p=1000$) using mouse study findings (172 genes) as external information.
  • Outcome: Tran-s-$\ell_0$ selected 4 genes from the mouse list and 0 from outside, whereas standard methods selected more genes from outside the list (potentially false positives).
  • Validation: The Tran-s-$\ell_0$ model achieved a lower Cross-Validated Mean Squared Error (CV-MSE) and higher pseudo-posterior inclusion probabilities for the mouse-derived genes compared to EBIC, suggesting the transfer of biological knowledge improved model quality.

5. Significance

This work bridges a critical gap between empirical practices in genomics and high-dimensional statistics and rigorous theoretical guarantees.

• Practical Impact: It provides a theoretically sound, computationally feasible method for leveraging auxiliary data (like animal models or GWAS summary statistics) to improve feature selection in target studies, particularly when sample sizes are small.
• Theoretical Advancement: It moves beyond parameter estimation (the focus of most transfer learning literature) to structural learning (variable selection), proving that structural information can be transferred even when the underlying data distributions (design matrices) differ significantly.
• Robustness: By proving robustness to negative transfer, the paper alleviates a major concern in applying transfer learning: the risk that irrelevant source data will degrade target performance.

In summary, the paper demonstrates that by encoding external information into block-specific $\ell_0$ penalties, one can break the "curse of dimensionality" more effectively than standard methods, achieving consistent variable selection in regimes where traditional approaches fail.