🤖 machine learning

Improving Improved Kernel PLS

This paper introduces optimized strategies for computing $\mathbf{X}$ rotations and $\mathbf{Y}$ loadings in Improved Kernel Partial Least Squares (IKPLS) algorithms that leverage modern hardware parallelism and mathematical equivalences to achieve significant speedups while preserving exact numerical results, all implemented in the open-source Python package `ikpls`.

Original authors: Ole-Christian Galbo Engstrøm

Published 2026-07-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ole-Christian Galbo Engstrøm

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a massive mystery, but instead of fingerprints, you have thousands of clues that are all jumbled together. In the world of data science, specifically a field called chemometrics, scientists often face this exact problem. They have a giant spreadsheet of "predictor" clues (like the chemical makeup of a substance) and a smaller set of "response" answers (like how strong a medicine is). To find the connection between the two, they use a mathematical tool called Partial Least Squares (PLS). Think of PLS as a super-smart sorting machine that tries to line up the messy clues with the answers to find the hidden pattern.

However, sorting through millions of clues takes a long time. The older, original way of doing this sorting is like trying to organize a library by picking up every single book one by one and asking, "Does this go here?" It's accurate, but it's painfully slow. A few years ago, scientists invented a faster version called "Improved Kernel PLS" (IKPLS), which is like using a conveyor belt to move books instead of carrying them. But even conveyor belts can have bottlenecks. This paper asks: "Can we make the conveyor belt move even faster without changing the books or the final arrangement?" The answer is yes, and the authors have found two clever tricks to speed things up significantly, especially when using modern super-fast computer chips.

The paper focuses on two specific steps in the IKPLS process that act like the engine of the sorting machine. The first step involves calculating "X rotations" (let's call this the R-step), which is essentially figuring out the best angle to tilt the data so the clues line up. The second step calculates "Y loadings" (the Q-step), which determines how much weight to give to each answer. The authors realized that the old way of doing these steps was like a single person trying to stack a tower of blocks one by one, waiting for each block to settle before adding the next. They proved that you can actually stack the whole tower in one giant, synchronized motion without changing the final shape of the tower at all.

For the R-step, the authors showed that instead of adding up numbers one by one (a slow, sequential process), you can use a "direct evaluation" strategy. Imagine a team of workers where, instead of passing a heavy box down a line, everyone grabs their part of the box and lifts it at the exact same time. The paper proves mathematically that this new method does exactly the same amount of math work as the old method, but because it lets modern computer processors (like the ones in your phone or a gaming PC) work in parallel, it finishes much faster. On a standard computer processor, this made the step about 2 times faster, but on a powerful graphics card (GPU), it sped up by a factor of up to 100 times in some cases!

The Q-step is where the magic gets even more interesting. The authors discovered a secret shortcut that allows the computer to skip a huge amount of work, but only under certain conditions. If the number of answers (Y) is small compared to the number of clues (X), or if there is only one answer to find, the old method was doing a lot of unnecessary calculations. It was like a chef chopping vegetables for a soup that only needs two carrots, but chopping a whole garden anyway. The authors proved that the chef can just grab the two carrots they already cut earlier in the process and skip the rest. This reduces the work from a massive amount to a tiny amount, making the calculation up to 100 times faster for those specific cases. However, they also showed that if the number of answers is huge (larger than the number of clues), this shortcut doesn't apply, and the computer has to do the full work anyway.

The authors didn't just guess these improvements; they proved them with rigorous math and then tested them on real computers. They built a free, open-source software package called ikpls that includes these new tricks. When they ran benchmarks, the results were clear: the new algorithms produced the exact same results as the old ones (no errors, no lost accuracy), but they finished the job much quicker. For a full data analysis, the new method made the process about 2 times faster on a standard computer and up to 6 times faster on a graphics card. The paper concludes that while the "R-step" improvement is a practical speed boost for everyone, the "Q-step" improvement is a game-changer for specific types of data problems, offering a massive reduction in the total work required.

In short, this paper is about taking a fast algorithm and making it even faster by reorganizing how the computer thinks about the math. It's a reminder that sometimes, the best way to go faster isn't to build a better engine, but to change how you drive the car. By proving that these new methods are mathematically identical to the old ones but computationally superior, the authors have given scientists a powerful new tool to analyze complex data without waiting around for the computer to catch up.

Problem
Improved Kernel Partial Least Squares (IKPLS) algorithms 1 and 2 are established as among the fastest PLS calibration methods, widely used in chemometrics for analyzing near-infrared (NIR) spectra in both regression and classification. While the original NIPALS algorithm is numerically stable but slow, IKPLS offers speed. However, this paper identifies that two specific computational steps within both IKPLS algorithms—calculating the X rotations ( $R$ ) and the Y loadings ( $Q$ )—can be further optimized. The goal is to accelerate these steps without altering the mathematical output, thereby improving the overall runtime of PLS fitting on modern hardware.

Methodology
The paper proposes two distinct algorithmic improvements, both proven to be mathematically equivalent to the original IKPLS steps:

Improved Computation of $R$ (X Rotations):
The original calculation of the rotation vector $r_a$ relies on a recurrence relation (Equation 1) that accumulates terms sequentially. The author replaces this with a direct evaluation strategy (Equation 2) using a matrix-vector product involving previously accumulated matrices $R_{a-1}$ and $P_{a-1}$ .
- Complexity: Both the original and new methods require the same number of multiplications ( $\Theta(K(a-1))$ ).
- Optimization: The new strategy is designed to parallelize better on modern multi-processor hardware, avoiding the sequential dependencies of term-by-term accumulation.
Improved Computation of $Q$ (Y Loadings):
The original calculation of the loading vector $q_a$ (Algorithm 2, line 8) involves a vector-matrix product with complexity $\Theta(KM)$ . The author identifies mathematical equivalences showing that $q_a$ can be derived from quantities already computed in earlier steps (specifically from the eigen-decomposition or vector operations in Step 2 of the IKPLS algorithm).
- Applicability: This improvement applies when $M=1$ (PLS1) or when $2 \le M < K$ (PLS2 where the response matrix has fewer columns than the predictor matrix). It does not apply when $2 \le M$ and $K \le M$ .
- Complexity: For applicable cases, the complexity is reduced from $\Theta(KM)$ to $\Theta(M)$ . This represents a reduction by a factor of $\Theta(K)$ .
- Proof: Theorem 1 proves that the new calculation (Algorithm 3) yields the exact same $q_a$ values as the original, ensuring that the subsequent deflation steps and final model parameters ( $W, P, Q, R, T$ ) remain identical.

Key Contributions

Algorithmic Reformulation: The paper provides a direct evaluation strategy for $R$ that, while asymptotically identical in operation count, is better suited for parallel execution.
Complexity Reduction: It presents a novel derivation for $Q$ that reduces the computational complexity from $\Theta(KM)$ to $\Theta(M)$ for specific common cases (PLS1 and "wide" PLS2). To the author's knowledge, this equivalence is identified for the first time.
Correctness Proofs: Rigorous proofs (Propositions 1–4 and Theorem 1) demonstrate that the improved steps produce exactly the same results as the original IKPLS algorithms.
Implementation: Both improvements are implemented in the free, open-source Python package ikpls, which also supports sample-weighted PLS and fast cross-validation.

Results
Benchmarks were conducted using NumPy (CPU) and JAX (GPU) across varying dataset sizes ( $N$ ), predictor counts ( $K$ ), response counts ( $M$ ), and component counts ( $A$ ).

Isolated Steps:
- The improved $R$ computation showed speedups of up to two orders of magnitude (e.g., 147x on GPU) compared to the term-by-term accumulation, with gains increasing as $A$ grows.
- The improved $Q$ computation showed speedups proportional to $K$ when $M=1$ or $M < K$ . For example, on the CPU, speedups reached 346x when $K=10^4$ and $M=1$ . When $M \ge K$ , the performance matched the original algorithm.
Full Fits:
- For entire model fits, the improvements yielded speedups of approximately 2x on CPU and 6x on GPU.
- The $R$ improvement contributed the larger share of the speedup in full fits, despite the $Q$ improvement having a lower asymptotic complexity, because the $R$ step is executed for every component and benefits significantly from parallelization.
- The relative speedup diminishes as $N$ increases (since other parts of the fit dominate) or when $M \ge K$ (where the $Q$ improvement is not applicable).

Significance
The paper claims that these improvements make IKPLS an even stronger choice for chemometricians requiring both calibration precision and speed.

Practical Impact: The improvements are "drop-in" replacements that produce identical results, allowing for immediate adoption in existing implementations.
Historical Context: The $R$ improvement validates a strategy already used (without proof) in the R package pls. The $Q$ improvement answers a question posed in the source code of pls since 2006 regarding the proportionality of loadings in PLS2, providing a formal proof and extending the solution to the PLS1 case.
Hardware Efficiency: The results highlight the importance of algorithm design that aligns with modern hardware capabilities (parallelism), demonstrating significant gains on both multi-core CPUs and GPUs.

More like this