Technical Summary: Anchor PCA

Problem Statement

Principal Component Analysis (PCA) is a standard technique for unsupervised dimensionality reduction, aiming to find a low-dimensional subspace that captures maximal variance. However, when data is collected across multiple related but heterogeneous domains (e.g., different geographic locations, time points, or experimental conditions), the principal subspaces often differ significantly.

Standard approaches, such as performing PCA on pooled data (poolPCA), maximize the average explained variance. The authors argue that this approach can be flawed: it may prioritize directions that exhibit high variance in only a few domains (spurious directions) while failing to capture directions that are robust and invariant across all domains. Consequently, poolPCA may yield embeddings that explain little variance in unseen target domains or fail to recover physically meaningful invariant structures. The paper addresses the need for a method that balances explained variance with the discovery of shared, invariant directions of variation across multiple domains.

Methodology: Anchor PCA

The authors propose Anchor PCA, a framework that explicitly trades off overall explained variance against the agreement between a shared low-rank embedding and the domain-specific low-rank embeddings. The method is defined in two variants:

1. Maximal Invariant Subspace ($S^\star$)

The authors first define the maximal invariant subspace $S^\star$ as the intersection of the top-$k$ principal subspaces of all $E$ source domains:
$$S^\star := \bigcap_{e=1}^E \text{Im}(\Pi^{(e)}_k)$$
where $\Pi^{(e)}_k$ is the projection matrix onto the top-$k$ eigenvectors of the covariance matrix $\Sigma_e$ for domain $e$.

2. AnchorPCA$_\infty$ (Hard Invariance)

To find a rank-$k$ subspace that strictly adheres to invariance, the authors define AnchorPCA$_\infty$. This method seeks a subspace that minimizes the disagreement (measured by the Frobenius norm of the difference between projection matrices) with all domain-specific top-$k$ subspaces, subject to maximizing the pooled explained variance among those solutions.
Formally, it solves:
$$W^{\infty}_k \in \arg \max_{W \in O_{p \times k}} \text{Tr}(W^\top \Sigma W) \quad \text{subject to} \quad W \in \arg \min_{\tilde{W}} \sum_{e=1}^E \| \Pi_{\tilde{W}} - \Pi^{(e)}_k \|_F^2$$
where $\Sigma$ is the average covariance and $\Pi_{\tilde{W}} = \tilde{W}\tilde{W}^\top$.

3. AnchorPCA$_\lambda$ (Soft Invariance)

Recognizing that exact invariance ($S^\star = \{0\}$) may not exist or may be too restrictive, the authors introduce AnchorPCA$_\lambda$ with a trade-off parameter $\lambda > 0$. This method relaxes the hard constraint into a soft penalty:
$$W^{\lambda}_k \in \arg \max_{W \in O_{p \times k}} \left( \text{Tr}(W^\top \Sigma W) - \lambda \sum_{e=1}^E \| \Pi_W - \Pi^{(e)}_k \|_F^2 \right)$$
This formulation is equivalent to performing standard PCA on a modified target matrix $M_\lambda$:
$$M_\lambda := \Sigma + 2E\lambda \Pi$$
where $\Pi = \frac{1}{E} \sum_{e=1}^E \Pi^{(e)}_k$ is the average projection matrix. This equivalence allows AnchorPCA$_\lambda$ to be solved efficiently using standard eigendecomposition.

4. Finite-Sample Implementation

For finite data, the authors propose empirical estimators. For AnchorPCA$_\infty$, a direct plug-in estimator is inconsistent due to estimation noise splitting eigenvalue blocks. They introduce a block-stabilized algorithm that groups consecutive eigenvalues of the empirical average projector $\hat{\Pi}$ within a tolerance $tol_n$ to recover the empirical invariant subspace. They also propose FindS$^\star$, a sequential testing procedure adapted from partial common principal component analysis (PCPCA) to estimate the dimension of the invariant subspace directly.

Key Theoretical Contributions

1. Recovery Guarantees:

   • Theorem 3.1: If a nontrivial maximal invariant subspace $S^\star$ exists, AnchorPCA$_\infty$ is guaranteed to contain it ($S^\star \subseteq \text{Im}(\Pi^{\infty}_k)$). Furthermore, as $\lambda \to \infty$, the leading directions of AnchorPCA$_\lambda$ converge to $S^\star$.
   • Consistency: Under mild conditions (finite fourth moments, strict eigengaps), the empirical estimators for both AnchorPCA$_\lambda$ and the block-stabilized AnchorPCA$_\infty$ are consistent as sample sizes grow.

2. Minimax Reconstruction Interpretation:

   • Theorem 4.1: AnchorPCA$_\lambda$ admits a minimax interpretation. It is the rank-$k$ projector that minimizes the worst-case average reconstruction error over a set of perturbed target domains where the covariance matrices undergo bounded, domain-specific inflations within the local top-$k$ eigenspaces. Specifically, it minimizes:
     $$\max_{(\Sigma'_1, \dots, \Sigma'_E) \in \mathcal{C}_\lambda} R(W; \Sigma'_1, \dots, \Sigma'_E)$$
     where $\mathcal{C}_\lambda$ represents covariances with inflations bounded by $2E\lambda \Pi^{(e)}_k$. AnchorPCA$_\infty$ corresponds to the limit where the slope of the worst-case error (the disagreement penalty) is minimized first.

Experimental Results

The authors validate Anchor PCA on both synthetic and real-world data:

1. Motivating Example: In a synthetic 4D example with three domains, poolPCA failed to recover the true invariant direction $b$ because it was dominated by a high-variance direction $w$ present in only one domain. Both AnchorPCA$_\lambda$ and AnchorPCA$_\infty$ successfully recovered the invariant subspace. AnchorPCA$_\lambda$ (with finite $\lambda$) balanced variance and invariance, while AnchorPCA$_\infty$ prioritized invariance.

2. Invariant Subspace Recovery: In simulations with random subspaces, the block-stabilized AnchorPCA$_\infty$ and FindS$^\star$ consistently recovered the true dimension and subspace of $S^\star$ as sample size increased. The method remained robust under non-Gaussian data distributions (Gaussian mixtures).

3. Gas Sensor Array Drift: Using real-world gas sensor data exhibiting temporal drift, the authors treated temporal batches as domains.

   • Setup: Source batches (B1–B6) were used for fitting; target batches (B7–B10) were held out.
   • Results: AnchorPCA$_\infty$ achieved the highest mean explained variance on target batches (87.3%) compared to poolPCA (75.5%) and a worst-case baseline (norm-maxRegret). AnchorPCA$_\lambda=1$ showed a trade-off, retaining more source variance but slightly lower target performance than the infinite penalty version. AnchorPCA$_\infty$ improved upon poolPCA on every held-out target batch.

Significance and Claims

The paper establishes Anchor PCA as a promising approach for robust unsupervised dimension reduction from multi-domain data. Its significance lies in:

• Theoretical Rigor: Providing formal guarantees for recovering maximal invariant subspaces and offering a minimax reconstruction interpretation under structured covariance inflations.
• Practical Utility: Demonstrating that focusing on shared directions of variation yields embeddings that generalize better to unseen domains compared to standard pooling or worst-case optimization baselines.
• Efficiency: The method reduces to a standard PCA problem on a modified matrix, making it computationally efficient.
• Connection to Causality: The approach draws inspiration from invariant learning and anchor regression, bridging the gap between causal invariance and unsupervised representation learning.

The authors note limitations, including sensitivity to near-eigenvalue ties and the assumption that perturbations are bounded inflations within local top-$k$ subspaces (not mean shifts or rotations). They also suggest future work on nonlinear extensions via autoencoders.