Homogeneity and Sub-homogeneity Pursuit: Iterative Complement Clustering PCA

1. Problem Statement

Principal Component Analysis (PCA) is the standard tool for dimensionality reduction in high-dimensional data. However, traditional PCA faces two critical limitations when data exhibits a group structure (e.g., stocks from different industries):

1. Failure to Identify Sub-homogeneity: PCA prioritizes directions of maximum variance, which usually correspond to homogeneity (common patterns shared across the entire dataset, such as a general market trend). Consequently, sub-homogeneity (group-specific patterns, such as industry-specific trends) often has lower variance and is treated as noise or ignored.
2. The "Curse of Dimensionality" in Clustering: When the number of variables ($p$) is large relative to the sample size ($n$), the common effect can dominate the covariance structure, masking the underlying cluster structure. Traditional clustering methods applied directly to the data often fail to separate groups because the shared homogeneity makes all variables appear highly correlated.
3. Circular Dependency: To accurately estimate homogeneity, one ideally needs to know the group structure. Conversely, to identify the group structure, one must first remove the dominant homogeneity. Traditional methods cannot resolve this circular dependency effectively.

2. Methodology: Iterative Complement-Clustering PCA (CPCA)

The authors propose a novel framework called CPCA to simultaneously estimate homogeneity (common components) and sub-homogeneity (cluster-specific components) through an iterative process.

A. Data Representation

The data $x_i$ is modeled as a low-rank representation:
$$x_i = \sum_{k=1}^{r_c} g_{ik}\phi_k + \sum_{j=1}^{J} \sum_{h=1}^{r_j} f_{ih}^{(j)} \gamma_h^{(j)} + \epsilon_i$$

• Homogeneity: Represented by common components $g_{ik}$ and eigenvectors $\phi_k$ shared across all groups.
• Sub-homogeneity: Represented by cluster-specific components $f_{ih}^{(j)}$ and eigenvectors $\gamma_h^{(j)}$ that are non-zero only for variables in cluster $j$.
• Structure: After removing common components, the residuals exhibit a block-diagonal covariance structure, where variables in different clusters are uncorrelated.

B. The CPCA Algorithm

The algorithm operates in three main stages:

1. Initial Step:

   • Perform standard PCA on the full dataset $X$ to estimate initial common components ($G_0, \Phi_0$).
   • Calculate the complement (residuals): $X^c_0 = X - G_0\Phi_0^\top$.
   • Apply Hierarchical Clustering on $X^c_0$ to obtain an initial grouping of variables.

2. Iterative Step (The Core Innovation):

   • Refine Common Components: Perform PCA within each current cluster, then combine these cluster-specific components and perform a second-layer PCA to extract updated common components ($G_s, \Phi_s$). This reduces the dimensionality burden of the "curse of dimensionality."
   • Refine Clusters (LOO-PCR Clustering): Update the cluster assignments using a Leave-One-Out Principal Component Regression (LOO-PCR) method.
     • For each variable $k$, leave it out.
     • Fit $J$ separate PCRs where the remaining variables in each cluster predict the left-out variable.
     • Assign the variable to the cluster that yields the minimum Sum of Squared Residuals (SSR).
     • If the minimum SSR exceeds a threshold, the variable is treated as a singleton cluster.
   • Repeat until the cluster assignments converge (measured by Adjusted Rand Index).

3. Final Step:

   • Perform the final estimation of common components and cluster-specific components using the converged clusters.

3. Key Contributions

1. Novel Framework (CPCA): Introduces an iterative approach that decouples the estimation of homogeneity and sub-homogeneity, solving the circular dependency problem.
2. LOO-PCR Clustering: Proposes a new clustering metric based on Principal Component Regression. Unlike correlation-based clustering, this method tests if a variable is well-predicted by the principal components of a specific group, making it more robust in high-dimensional settings ($p > n$).
3. Theoretical Guarantees: The paper provides theoretical proofs (Theorem 1) showing that if sub-homogeneity across clusters is distinct, the LOO-PCR method can consistently identify the correct cluster membership.
4. Dimensionality Reduction Strategy: By clustering first and then estimating common components at the group level, the method alleviates the curse of dimensionality, leading to more accurate estimation of the common factors compared to global PCA.

4. Results

The authors validate CPCA through extensive simulations and a real-world stock return analysis.

Simulation Studies

• Recovery Accuracy: CPCA (specifically the final iterative version, CPCA-F) significantly outperforms traditional PCA and non-iterative CPCA (CPCA-I) in recovering the original data (lower Mean Squared Recovering Error - MSRE).
• Small Sample/Large Dimension: In scenarios where $n < p$ (e.g., $n=30, p=100$), traditional PCA fails to identify sub-homogeneity, while CPCA successfully recovers both common and group-specific structures.
• Clustering Performance: The LOO-PCR clustering method achieves higher Adjusted Rand Indices (ARI) than hierarchical clustering, particularly when the common effect masks the cluster structure.

Applications

1. Principal Component Regression (PCR):
   • In predicting responses where only specific group components matter, CPCA-based PCR (combined with Group Lasso) significantly reduces Mean Squared Prediction Error (MSPE) compared to standard PCA.
   • It successfully identifies that small-variance, group-specific components can be critical predictors, which standard PCA misses.
2. Covariance Matrix Estimation:
   • CPCA provides a more accurate estimate of the covariance matrix ($\Sigma$) and precision matrix than POET (Principal Orthogonal complEment Thresholding) and standard PCA, as it captures both low-rank (common) and block-diagonal (group-specific) structures.
3. Real Data Analysis (Stock Returns):
   • Dataset: 160 stocks from 8 industries (2014).
   • Clustering: Traditional PCA clustering yielded an ARI of 0.09 (failed to separate industries). CPCA-F improved this to 0.72, correctly identifying industry-specific clusters after removing the market effect.
   • Portfolio Construction: Using the CPCA-estimated covariance matrix to construct a Global Minimum Variance Portfolio (MVP) resulted in the lowest Standard Deviation (0.008) and the highest Sharpe Ratio (0.143) compared to portfolios built using traditional PCA or POET.

5. Significance

This paper addresses a fundamental gap in high-dimensional data analysis: the inability of standard dimension reduction techniques to capture hierarchical heterogeneity.

• Theoretical Impact: It formalizes the concept of "sub-homogeneity" and provides a rigorous iterative algorithm to estimate it.
• Practical Impact: The method is highly relevant for financial modeling (where market vs. industry effects must be separated), genomics (gene pathways), and any field where data comes from distinct sub-populations that share a global trend.
• Robustness: The proposed LOO-PCR clustering is particularly valuable in "small $n$, large $p$" regimes where traditional correlation-based clustering fails.

In summary, CPCA offers a superior alternative to PCA for structured high-dimensional data, enabling researchers to uncover hidden group-specific patterns that are otherwise obscured by dominant global trends.