🤖 machine learning

TreeCCA: Canonical Correlation Analysis via Gradient-Boosted Trees

The paper introduces TreeCCA, a novel method that trains gradient-boosted tree ensembles end-to-end as canonical correlation analysis encoders using a custom Eckart-Young loss, thereby achieving state-of-the-art nonlinear correlation extraction and interpretability with the plug-and-play reliability and efficiency of standard tree-based libraries.

Original authors: James Chapman

Published 2026-07-30

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: James Chapman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but you have two different notebooks filled with clues. One notebook lists the suspects' heights, weights, and shoe sizes, while the other lists their favorite foods, music genres, and hobbies. Your job is to find the hidden connection between these two lists. Maybe the taller suspects tend to like jazz, or perhaps people who love spicy food also prefer hiking. This is the heart of a statistical detective game called Canonical Correlation Analysis (CCA). It's a tool scientists use to find the strongest links between two different sets of data, helping them understand how different parts of the world—like genes and diseases, or brain activity and behavior—talk to each other.

For a long time, detectives used simple, straight-line rulers to find these connections. They assumed the relationship was as straightforward as "taller equals jazz-lover." But the real world is messy and curvy; sometimes the connection is a twisty, complex curve that a straight ruler just can't measure. To fix this, scientists started using "Deep Learning," which is like hiring a team of super-smart, flexible robots to find those curvy patterns. These robots are powerful, but they are also like black boxes: they are hard to tune, they can get confused by small amounts of data, and once they find an answer, it's often impossible to see why they made that choice. They just give you the answer without showing their work.

This is where a new paper steps in with a fresh idea. The authors, led by James Chapman, ask: "What if we used the tools that are already the kings of tabular data—Gradient-Boosted Trees—to solve this mystery?" You might know these trees as the engines behind XGBoost and LightGBM, the tools that win almost every competition for predicting things like house prices or loan risks. They are famous for being reliable, easy to use, and, most importantly, transparent. You can look at a tree and see exactly which clue mattered most. The paper introduces TreeCCA, a method that trains these tree-based engines to find the complex, curvy connections between two sets of data, just like the fancy robots do, but with the reliability and clarity of a well-built tree.

The Tree That Learns to Correlate

The paper proposes TreeCCA, the first method to train gradient-boosted tree ensembles as "encoders" for CCA. Think of an encoder as a translator that takes raw data (like a list of numbers) and turns it into a secret code (an embedding) that highlights the most important relationships. Usually, these translators are neural networks (the robots). TreeCCA replaces the robots with a forest of decision trees.

The secret sauce that makes this possible is something called the Eckart-Young (EY) loss. In the past, training trees for this job was like trying to teach a dog to do calculus; the math didn't fit. The EY loss, however, provides a special set of instructions (gradients) that tell the trees exactly how to adjust their branches to get better at finding correlations. It's like giving the trees a map that says, "If you split here, you get closer to the truth." Because these instructions are so clear, they can be plugged directly into standard tree libraries like XGBoost or LightGBM without needing to rewrite the software.

Why Trees Might Be Better Than Robots

The authors tested TreeCCA against the current champions: Deep CCA (the robot approach) and Linear CCA (the straight-line ruler). The results were surprisingly good, and in some cases, trees won the race.

On synthetic tests designed to be tricky, TreeCCA found stronger connections than the robots. For example, on a test called "Signed Power," TreeCCA scored 2.61, beating Deep CCA's 2.43. On another test called "Hermite," where the connection was so complex that a straight ruler couldn't see it at all (scoring near zero), TreeCCA found a strong signal with a score of 2.93, slightly edging out Deep CCA's 2.89.

But the real magic happens when the data gets huge or messy. In a test using a massive dataset of handwritten digits (Split MNIST with 54,000 images), the robot approach (Deep CCA) started to memorize the training data instead of learning the rules, leading to a huge gap between its practice score and its test score (a ratio of 1.95). TreeCCA, however, stayed calm and consistent, with a ratio of just 1.04. It seems the trees are naturally better at not overthinking things, making them more reliable for real-world data where you don't have millions of examples.

The Superpower of "Why"

Perhaps the most exciting part of TreeCCA isn't just that it works, but that it explains how it works. Neural networks are often criticized for being opaque; you can't easily tell which feature they used to make a decision. Trees, on the other hand, are built on splits: "Is the temperature above 70? Yes/No." This structure gives them native interpretability.

The authors demonstrated this on the UCI HAR dataset, which tracks human movement using smartphone sensors (accelerometers and gyroscopes). They wanted to see if the model could figure out that the magnitude of movement (how hard the phone is spinning) was the key to identifying activities. TreeCCA didn't just get the right answer; its "feature importance" scores clearly showed that the magnitude features were the most important, perfectly matching a physics hypothesis about how spinning works. A neural network gave the right answer but couldn't explain why, leaving the "why" hidden in its millions of parameters. TreeCCA laid the reasoning out on the table.

Finding Signals in the Noise

The paper also tackled a specific problem where other methods fail: sparse signals. Imagine you have 500 clues, but only 5 of them actually matter, and the connection between them is non-linear (curvy). A method called PMD (a popular sparse CCA tool) relies on linear math, so it gets completely confused by this type of signal, performing no better than random guessing. TreeCCA, however, treated the problem like a game of "20 Questions." It asked, "Which feature should I split on?" and naturally ignored the 495 useless noise features. On a test with 50 features, TreeCCA achieved perfect precision (1.00), correctly identifying the 5 true clues every time, while PMD found nothing.

The Bottom Line

TreeCCA isn't just a new algorithm; it's a shift in perspective. It suggests that for many problems involving tabular data (rows and columns of numbers), we don't need to reach for the complex, hard-to-tune neural networks to find deep connections. Instead, we can use the robust, reliable, and explainable trees that are already dominating the field.

The authors found that TreeCCA matches or beats Deep CCA in accuracy, runs 5 times faster on some benchmarks, and provides a clear explanation of its reasoning. While the paper notes that formal mathematical proofs for how the trees converge are still being worked out, the simulations and real-world tests are strong. It opens the door for a new kind of "self-supervised learning" where trees can learn from data without labels, potentially solving problems that linear methods simply cannot see. In the world of data science, TreeCCA is a reminder that sometimes the best tool isn't the most complex one, but the one that knows exactly which branch to follow.

Technical Summary: TreeCCA

Problem Statement

Canonical Correlation Analysis (CCA) is a fundamental technique for finding projections of two matched views that maximize cross-view correlations. While CCA has been successfully applied in domains like multi-omics to uncover associations between genomics, transcriptomics, and proteomics, existing methods face significant limitations:

Linearity: Classical CCA relies on linear projections, failing to capture nonlinear relationships.
Neural Limitations: Deep CCA (DCCA) addresses nonlinearity using neural encoders but frequently underperforms gradient-boosted trees (GBTs) on tabular data due to the latter's superior handling of threshold structures, feature interactions, and moderate sample sizes.
Sparse Blindness: Sparse CCA methods (e.g., PMD) use L1 penalties to select features but remain structurally blind to signals with zero linear cross-view covariance.
Interpretability: Neural encoders lack native interpretability, making it difficult to understand which features drive cross-view correlations.

There is currently no method that utilizes gradient-boosted trees as CCA encoders, despite GBTs dominating tabular machine learning benchmarks.

Methodology: TreeCCA

The authors propose TreeCCA, the first method to train gradient-boosted tree ensembles end-to-end as CCA encoders. The core technical enabler is the Eckart-Young (EY) loss, which reformulates CCA as an unconstrained minimization problem over embedding matrices $Z_1$ and $Z_2$ , bypassing the need for generalized eigenvalue problems.

Key Technical Components

EY Loss and Gradients:
The EY loss is defined as:
$L_{EY}(Z_1, Z_2) = -2 \text{tr}(C) + \|V\|_F^2$
where $C$ is the symmetrized cross-covariance and $V$ is the within-view covariance. Crucially, this formulation provides closed-form per-sample gradients:
$\frac{\partial L_{EY}}{\partial Z_{1,i}} = \frac{4}{N-1} (-Z_{2c,i} + V Z_{1c,i})$
These gradients depend only on the empirical embeddings, not on the encoder architecture. This allows the gradients to be supplied directly to any standard GBT library (XGBoost, LightGBM) via their custom objective API without modifying the library itself.
Training Procedure:
- Two-Encoder Setup: Two boosters, $B_1$ and $B_2$ , are trained to map views $X_1$ and $X_2$ to embeddings $Z_1$ and $Z_2$ .
- Incremental Embedding Cache: To avoid the $O(R^2NK)$ cost of re-evaluating all trees at every round, the method maintains a cached embedding $Z$ and updates it incrementally by evaluating only the newly added tree. This yields a 50–100× speedup.
- Unit Hessians: The method sets the Hessian $h_i = 1$ for all samples. Using the true EY Hessian would cause the denominator in the leaf weight calculation to be dominated by the L2 regularization term $\lambda$ , leading to divergence. Unit Hessians ensure the leaf value equals the average gradient, providing stability.
- Gradient Normalization: Raw EY gradients scale with $O(1/\sqrt{N})$ . The method normalizes gradients to a fixed target standard deviation ( $\sigma_{target} = 0.1$ ) to decouple training speed from dataset size and ensure uniform interpretability of the learning rate.
Extensions:
- TreeMCCA: Generalizes to $M > 2$ views by summing pairwise EY objectives.
- TreeCCA-SSL: A siamese single-encoder variant for self-supervised learning, where a shared booster is trained on independently augmented views.

Key Contributions

First GBT-based CCA: Demonstrates that the EY loss enables training GBTs as CCA encoders with no library modifications, leveraging the global optimization landscape of the EY loss (no spurious local minima).
Efficient Training: Introduces an incremental embedding cache and principled unit Hessians, achieving significant speedups and stability.
Nonlinear Feature Recovery: Achieves perfect precision in recovering sparse, nonlinear signals where linear cross-view covariance is zero—a regime where PMD fails completely.
Native Interpretability: Unlike neural encoders, TreeCCA provides gain importances that reveal which specific features drive cross-view correlations without extra cost.
Scalability and Generalization: Demonstrates superior generalization on large-scale datasets (Split MNIST) compared to Deep CCA, avoiding overfitting through the inherent regularization of GBTs.

Experimental Results

The paper evaluates TreeCCA across synthetic and real-world benchmarks:

Synthetic Benchmarks (Signed Power & Hermite):
- TreeCCA outperforms Deep CCA and Kernel CCA (KCCA) in Total Correlation Captured (TCC).
- On the Hermite benchmark (zero linear covariance), TreeCCA achieves 2.93 TCC vs. 2.89 for Deep CCA and near-zero for Linear CCA.
- Training time is significantly lower than Deep CCA (~50s vs. ~300s) and avoids the $O(N^2)$ memory of KCCA.
Scale (Split MNIST):
- On a dataset with $N=54,000$ and $K=50$ , Deep CCA overfits (Train/Val TCC ratio = 1.95×), while TreeCCA generalizes effectively (ratio = 1.04×).
Sensor Fusion (UCI HAR):
- TreeCCA achieves comparable accuracy to Deep CCA (0.466 vs. 0.510 probe accuracy) at 5× lower wall-clock cost (17s vs. 80s).
- Interpretability: XGBoost gain importances validated a physics-motivated hypothesis: magnitude features (related to angular velocity) were the primary drivers of correlation in the gyroscope view, a structural insight not readily available with neural encoders.
Sparse Recovery:
- On a synthetic benchmark with zero linear cross-view covariance, TreeCCA achieved Precision@S = 1.00 at $p=50$ , while PMD remained at random baseline.
Multi-View (TreeMCCA):
- Evaluated on five real-world multi-view datasets (e.g., Caltech101-7, 3Sources). TreeMCCA consistently matched or exceeded Linear CCA in TCC and downstream classification accuracy, with the largest gains observed where cross-view signals were highly nonlinear.
Self-Supervised Learning (TreeCCA-SSL):
- On a synthetic benchmark where the signal is invisible to any affine encoder (due to sign interactions), TreeCCA-SSL achieved 0.630 accuracy (3.8× above random), whereas PCA failed at random chance (0.159).

Significance and Claims

The paper claims that TreeCCA offers a unique combination of properties previously unavailable in a single method:

Nonlinear and Sparse Signal Recovery: It matches or exceeds Deep CCA on synthetic benchmarks and generalizes better at scale, while recovering signals that linear sparse methods structurally cannot.
Interpretability: It provides native feature diagnostics (gain importances) that allow practitioners to validate hypotheses about the data (e.g., physics-based signal structures) directly from the model.
Practical Breadth: It achieves comparable performance to Deep CCA with significantly lower computational cost and no need for architecture search or hyperparameter tuning beyond standard GBT defaults.

The authors position TreeCCA as a natural first step for practitioners starting multi-view analyses, particularly in fields like multi-omics and clinical applications where sample sizes are moderate, data is tabular, and interpretability is critical. The paper acknowledges limitations, including the degradation of sparse recovery precision at very high $p/N$ ratios and the lack of formal convergence theory for the alternating GBT update procedure.