⚡ electrical engineering

Reframing preprocessing selection as model-internal calibration in near-infrared spectroscopy: A large-scale benchmark of operator-adaptive PLS and Ridge models

This paper introduces an operator-adaptive calibration framework that integrates spectral preprocessing selection directly into linear regression models (PLS and Ridge) for near-infrared spectroscopy, demonstrating through a large-scale benchmark that this approach achieves superior or comparable predictive accuracy to conventional methods while significantly reducing computational costs, ensuring auditability, and preserving model interpretability.

Original authors: Gregory Beurier, Robin Reiter, Camille Noûs, Lauriane Rouan, Denis Cornet

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Gregory Beurier, Robin Reiter, Camille Noûs, Lauriane Rouan, Denis Cornet

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to identify different types of fruit just by looking at how they reflect light (this is what Near-Infrared Spectroscopy or NIRS does). The robot needs a "recipe" (a mathematical model) to make the right guess.

For a long time, the biggest problem wasn't the robot's brain; it was the preparation of the data. Before feeding the data to the robot, scientists had to manually "clean" the light signals. They had to decide: Should I smooth out the bumps? Should I remove the background noise? Should I sharpen the edges?

The Old Way: The "Guess-and-Check" Kitchen

Traditionally, scientists treated this cleaning step like a massive, expensive game of "Guess and Check."

They would try thousands of different cleaning recipes (pipelines).
They would test each one to see which worked best.
The Problem: This took a huge amount of computer time. It was unstable (if you tried it again with slightly different data, you might pick a totally different recipe). And it was hard to explain why a specific recipe was chosen. It was like a chef saying, "I added salt because it tasted good," without being able to write down the exact amount.

The New Way: The "Smart, Self-Cleaning" Robot

This paper introduces a new method called Operator-Adaptive Calibration. Instead of treating the cleaning steps as a separate, external game, the authors built the cleaning choices inside the robot's brain itself.

Think of it like this:

The Old Robot: You hand it a messy pile of fruit. You spend hours manually washing, peeling, and slicing every single piece before giving it to the robot. If you change the washing method, you have to start the whole process over.
The New Robot (Operator-Adaptive): You hand it the messy pile, but the robot has a built-in "toolbox" of cleaning methods (smoothing, sharpening, etc.). As it learns, it automatically picks the best tool for the job while it's learning.

How It Works (The Magic Tricks)

The paper describes two main ways this new robot learns, using two different mathematical "brains":

The PLS Brain (The Covariance Detective):
This brain looks for patterns between the light and the fruit type. The authors found a mathematical shortcut (an "identity") that lets the robot try different cleaning tools without actually having to re-calculate the whole dataset every time. It's like having a magic lens that lets you see what the fruit would look like if it were cleaned, without actually touching it. This makes the process incredibly fast (seconds instead of hours).
The Ridge Brain (The Geometry Architect):
This brain looks at the shape of the data. It treats the cleaning tools as different ways to stretch or shrink the data's geometry. It builds a "map" of all possible cleaning styles and picks the one that makes the map most stable.

The "Branch" Trick for Tricky Problems

Some cleaning methods (like removing specific types of noise) are too complex to be simple "tools" in the toolbox. They depend on the specific sample being looked at.

The authors put these tricky methods in a special "Branch" (like a side door).
The robot tries the main toolbox first. If the data needs that special side-door treatment, it uses it. But it does this carefully so it doesn't "cheat" by peeking at the test answers while it's learning.

What Did They Find?

The authors tested this new method on 50+ different real-world datasets (from food to plants to fuel).

Better Results: The new method was often more accurate than the old "Guess-and-Check" method. In fact, the new "PLS" robot beat the old standard in 42 out of 57 cases.
Much Faster: The new method didn't need to run thousands of tests. It found a great solution in seconds.
Transparent: Because the cleaning steps are part of the model, you can look at the final robot and say, "Ah, it chose to smooth the data and remove the baseline trend." You have a clear log of why it made that choice.

The Big Takeaway

This paper argues that we should stop treating data cleaning as a separate, messy chore. Instead, we should bake the choices directly into the model.

The Analogy:

Old Way: You hire a team of 5,000 chefs to try different ways of chopping vegetables, pick the best one, and then hire a new team to cook the meal.
New Way: You hire one super-chef who has a built-in instinct for the best way to chop vegetables while they are cooking. It's faster, cheaper, and you know exactly how they did it.

The result is a system that is faster to build, easier to trust, and just as (or more) accurate than the old, complicated way of doing things.

Technical Summary: Operator-Adaptive Calibration in Near-Infrared Spectroscopy

Problem Statement
In routine Near-Infrared Spectroscopy (NIRS) workflows, the development of reliable calibration models is often bottlenecked by the selection of spectral preprocessing. Traditionally, this selection is performed via external pipeline searches (e.g., testing combinations of smoothing, derivatives, scatter correction, and baseline correction) across large hyperparameter spaces. The authors argue that this approach is computationally expensive, statistically unstable on small calibration sets, and difficult to audit. Furthermore, the final model's performance becomes dependent on external choices that are opaque and hard to trace, creating a "winner's curse" where selected pipelines may reflect validation noise rather than true signal.

Methodology: Operator-Adaptive Calibration
The paper introduces operator-adaptive calibration, a paradigm shift that reframes preprocessing selection as an internal, auditable component of the calibration model itself, rather than an external search.

Framework: The approach treats linear spectral treatments as a bank of linear operators $\{A_1, \dots, A_m\}$ acting on the spectra. The calibration model internally selects the optimal operator(s).
Linear vs. Nonlinear Handling:
- Strict Linear Operators: Treatments like Savitzky–Golay smoothing, derivatives, and polynomial detrending are encoded as fixed matrices. These are integrated directly into the model's algebra.
- Fold-Local Branches: Nonlinear or sample-adaptive corrections (e.g., SNV, MSC, ASLS) are handled as explicit "fold-local" branches. This prevents data leakage by ensuring these corrections are fitted only within cross-validation folds, preserving the distinction between linear operator algebra and fitted preprocessing.
Model Instantiations:
- AOM-PLS (Partial Least Squares): Utilizes covariance identities to implement operator selection efficiently. Specifically, it leverages the identity $(XA_b^T)^T Y = A_b X^T Y$ to compute cross-covariances without materializing transformed spectra. This allows for fast NIPALS and SIMPLS variants that preserve original-wavelength coefficients, ensuring interpretability.
- AOM-Ridge (Ridge Regression): Formulates the problem in the dual space using operator-specific sample kernels. The method constructs a kernel as a sum of operator-induced geometries ( $K = \sum s_b^2 X_c A_b^T A_b X_c^T$ ). Like PLS, it recovers coefficients in the original spectral space, maintaining interpretability.

Key Contributions

Paradigm Shift: Proposes treating preprocessing alternatives as structured, recoverable model components rather than competing black-box pipelines.
Efficient Algorithms: Provides fast implementations for PLS and Ridge that retain original-space coefficients, enabling interpretability and deployment without sacrificing the benefits of adaptive selection.
Large-Scale Validation: Evaluates the framework on a heterogeneous benchmark of over 50 NIRS datasets covering diverse domains (food, agriculture, plant phenotyping, fuels) and sample sizes.
Traceability: Demonstrates a workflow where operator choices are logged and auditable, reducing the complexity of calibration search campaigns.

Results
The study compares operator-adaptive models against conventional PLS, Ridge, CatBoost, and CNN baselines under documented search budgets.

AOM-PLS: A compact variant combining a small operator bank (identity, smoothing, derivatives, detrending) with an ASLS branch achieved a median RMSEP/PLS ratio of 0.960 across 57 datasets, winning on 42 datasets. Crucially, this was achieved with a median fit time of ~1.36 seconds, replacing thousands of external hyperparameter search evaluations.
AOM-Ridge: A deployable "Blender" selector improved over tuned Ridge by a median 2.22% (35 wins on 52 datasets).
Search Budget: The proposed method drastically reduces the "cost" of calibration. While reference baselines (PLS-HPO, Ridge-HPO) required approximately 5,400 candidate evaluations per dataset, the compact AOM-PLS approach operates within a structured, small operator bank, effectively replacing the external search with a single linear model fit.
Robustness: The study found that larger operator banks did not systematically improve external test performance and often increased variance. Compact banks provided a better bias-variance tradeoff for routine calibration.

Significance and Claims
The paper claims that operator-adaptive calibration offers a practical route toward faster, more robust, and auditable NIRS method development.

Interpretability: Unlike deep learning or complex pipeline searches, the resulting models remain linear maps from original spectra to responses, with coefficients that can be inspected.
Auditability: The selection of operators is logged within the model, making the calibration object transparent and easier to transfer to non-expert users.
Efficiency: By moving preprocessing selection inside the model, the method eliminates the need for massive external hyperparameter optimization campaigns, making routine recalibration and model updates more feasible.

The authors emphasize that this is not merely a new preprocessing filter, but a change in the calibration paradigm: preprocessing is no longer a pre-processing step but a core, selectable component of the analytical calibration object. The work is implemented in the open-source nirs4all Python library and its graphical interface, nirs4all Studio.