Technical Summary: PLS in the Mirror of Self-Attention

Problem Statement

The paper addresses the theoretical disconnect between Partial Least Squares (PLS), a classical statistical method for handling high collinearity in multivariate data, and the modern neural network paradigm, specifically the Transformer architecture. While PLS is traditionally formulated as a constrained maximization of cross-covariance between latent projections of predictors ($X$) and responses ($Y$), the author seeks to reframe PLS within the language of self-attention mechanisms. The core problem is to demonstrate that PLS can be viewed as a linearized form of self-attention, thereby allowing PLS to be studied through the lens of deep learning, while simultaneously suggesting that the dimensionality reduction inherent in PLS may offer insights into how self-attention normalizes dimensions for improved learning.

Methodology

The paper employs a mathematical reformulation approach to bridge the gap between PLS and Transformer architectures:

1. Reformulation of PLS as Regression:
   Traditionally, PLS is defined by maximizing the trace of the cross-covariance between projected matrices $T$ and $U$ (Equation 2). The author proposes an alternative cost function (Equation 13) that frames PLS as a regression problem minimizing the squared error between the transformed predictors and the transformed responses:
   $$\text{argmin}_{P,D,Q} \sum_{(X,Y) \in S} |XPD - YQ|^2$$
   subject to orthogonality constraints ($P^T P = I, Q^T Q = I$). Here, $XP$ represents the projection of predictors, and $TDQ^T$ represents the transformation to predict the response.

2. Integration of Reconstruction Error:
   To balance latent variable extraction with regression accuracy, the author introduces a generalized loss function (Equation 14) that includes parameterized reconstruction errors for both $X$ and $Y$:
   $$L(P, Q, D; \alpha, \beta) = \frac{1}{2} \left[ |XPD - YQ|^2 + \alpha |XPP^T - X|^2 + \beta |YQQ^T - Y|^2 \right]$$
   This formulation allows for the optimization of PLS parameters using gradient descent methods, similar to training neural networks.

3. Mapping to Self-Attention:
   The paper draws a structural parallel between the PLS decomposition and the Self-Attention mechanism in Transformers.

   • PLS: Decomposes $X$ and $Y$ into projections $T=XP$ and $U=YQ$ to maximize covariance.
   • Self-Attention: Projects input $X$ into Query ($Q$), Key ($K$), and Value ($V$) spaces via weight matrices ($W_Q, W_K, W_V$) and computes attention via $\text{softmax}(QK^T/\sqrt{l})V$.
   • The Connection: The author posits that the PLS projection matrices ($P, Q$) correspond to the weight matrices in the attention mechanism. Specifically, the regression step in PLS (predicting $Y$ from $X$ via latent components) is shown to be structurally equivalent to the decoder block in a Transformer, where the source features ($X_f$) and target features ($Y$) interact via attention.

Key Contributions

• Linearized Self-Attention Interpretation: The paper provides an observation that casts PLS as a linearized self-attention mechanism. This reframing allows PLS to be analyzed within the neural network paradigm, suggesting that the "loading matrices" in PLS function similarly to the weight matrices in attention layers.
• Regression-Based PLS Formulation: By shifting the objective from cross-covariance maximization to explicit squared-error minimization (Equation 13), the paper offers a formulation that is more flexible, potentially accommodating non-orthogonal transformations and nonlinear activations, and is directly solvable via gradient descent.
• Dimensionality Normalization Hypothesis: The author suggests that the dimensionality reduction and predictor selection inherent in PLS may indicate that self-attention mechanisms include a degree of dimensionality normalization that contributes to improved learning, drawing a parallel between the two methodologies.

Results and Claims

The paper does not present empirical experimental results, datasets, or performance benchmarks. Instead, the "results" are theoretical observations and mathematical equivalences:

• Mathematical Equivalence: The author demonstrates that minimizing the squared error $|WX - Y|^2$ is mathematically equivalent to maximizing the dot product $(WX) \cdot Y$, thereby linking the regression-based PLS formulation (Equation 13) to the covariance-based formulation (Equation 2).
• Structural Alignment: The paper claims that the decoder block in a Transformer, which mixes source and target attention, is symbolically equivalent to the regression step in PLS (Equation 5).
• Optimization Path: The reformulated loss function (Equation 14) is claimed to be solvable via gradient descent, unlike the traditional iterative algorithms (like NIPALS) often used for PLS, thus aligning PLS with standard deep learning optimization techniques.

Significance and Scope

The paper positions itself as a "note" providing an "interesting observation" rather than a comprehensive study with new empirical findings. Its significance lies in the conceptual unification of a classical statistical technique (PLS) with modern deep learning architectures (Transformers).

The author modestly claims that this perspective allows PLS to be "studied within the neural network paradigm" and suggests that the properties of PLS (dimensionality reduction) may offer insights into the mechanics of self-attention. The paper explicitly avoids inventing new applications or future implications, focusing strictly on the mathematical mapping between the two frameworks. It serves as a theoretical bridge, proposing that the dimensionality normalization found in PLS is a feature potentially present in self-attention mechanisms, thereby offering a new lens for understanding how Transformers process multivariate data.