🤖 machine learning

Spectroscopy Analysis with Machine Learning Regression for the Quantification of Carbon and Nitrogen Contents in Inceptisol and Oxisol Soil Types: Comparing Different Preprocessing and Validation methods as well as Feature Importance

This study demonstrates that a stacked ensemble machine learning model, optimized with Savitzky-Golay filtering and NIPALS-based outlier removal, effectively quantifies carbon and nitrogen in Oxisol and Inceptisol soils using portable NIR spectroscopy, achieving high predictive accuracy (RPD > 2.0) for rapid, sustainable agricultural decision-making.

Original authors: Vinicius Herique Kieling, Guilherme Macedo Baggio, Felipe Augusto Bueno Rossi, Marco Antonio de Castro Barbosa, Dalcimar Casanova, Larissa Macedo dos Santos Tonial, Jefferson Tales Oliva

Published 2026-07-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Vinicius Herique Kieling, Guilherme Macedo Baggio, Felipe Augusto Bueno Rossi, Marco Antonio de Castro Barbosa, Dalcimar Casanova, Larissa Macedo dos Santos Tonial, Jefferson Tales Oliva

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a farmer trying to figure out how healthy your soil is. Traditionally, to check if your soil has enough Carbon (which feeds plants) and Nitrogen (which helps them grow), you'd have to send a dirt sample to a lab. There, scientists would mix it with strong, smelly chemicals, boil it, and wait days for results. It's slow, expensive, and a bit dangerous.

This paper is about a faster, cleaner way to do this: using a "high-tech flashlight" called Near-Infrared (NIR) spectroscopy combined with Machine Learning (smart computer programs).

Here is the story of what they did, explained simply:

1. The "Magic Flashlight" vs. The "Old Lab"

Think of the traditional lab method as a slow, manual detective who has to dig through every single clue one by one. The NIR method is like a super-fast scanner. You just shine a special light on the soil, and the soil "sings" back a unique song (a spectrum) based on what's inside it.

However, this "song" is incredibly complex. It's like listening to a symphony orchestra where every instrument is playing at once. A human ear (or a human brain) can't make sense of it. That's where the Machine Learning comes in. It's the conductor that listens to the chaos and figures out exactly how much Carbon and Nitrogen is in the mix.

2. The Two Types of Soil (The "Characters")

The researchers tested this on two very different types of soil, like testing a new recipe on two different kinds of dough:

Oxisol: Think of this as the "old, weathered" soil. It's common in Brazil, very deep, and has been around for a long time. It's like a well-worn leather jacket.
Inceptisol: This is the "young, developing" soil. It's shallower and less formed, like a brand-new pair of jeans that hasn't been washed yet.

3. Cleaning the Signal (The "Noise" Problem)

When the flashlight shines on the soil, the signal isn't perfect. Sometimes the machine gets a little static, or the soil sample has a weird shape that confuses the light. This is like trying to hear a song while someone is clapping loudly in the background.

The researchers tried many ways to "clean up" the audio:

Smoothing (Savitzky-Golay): Like using noise-canceling headphones to smooth out the static.
Removing Outliers: Like muting the person who is clapping too loudly.
The Winner: They found that combining the "smoothing" with "removing the loud clappers" gave the clearest song.

4. Training the Computer (The "Student")

Once the signal was clean, they had to teach the computer how to read it. They used a technique called Stacking, which is like forming a study group.

Instead of just one student (one algorithm) trying to solve the problem, they had a team: a "math whiz" (PLS), a "pattern finder" (SVR), and a "rule follower" (Ridge).
They let each student guess the answer, and then a "teacher" (a meta-model) looked at all their guesses to make the final, best decision.

5. The Results: Who Did Better?

The results were a bit like a sports match between two different teams:

The Oxisol Team (The Old Soil): They were the champions! The computer model was incredibly accurate. It could predict the Carbon and Nitrogen levels with a "score" (R²) of about 0.91 and 0.89. This means the computer was almost as good as the slow, chemical lab method, but it took seconds instead of days.
The Inceptisol Team (The Young Soil): They did a "good job," but not as perfectly. The scores were lower (around 0.77 to 0.79). The researchers think this is because the young soil is more variable and "messy," making it harder for the computer to find a clear pattern.

6. The "Secret Ingredients" (Wavelengths)

The researchers also asked: Which parts of the light spectrum were actually doing the work?
They found that the "middle" part of the light spectrum (around 1300–1400 nanometers) was the most important. This is the "sweet spot" where the soil's Carbon and Nitrogen "sing" the loudest. The ends of the spectrum (near 1600–1700 nm) were mostly just noise, like static on a radio.

The Bottom Line

This paper proves that you don't need a lab full of chemicals and days of waiting to know if your soil is healthy. By using a portable light scanner and smart computer programs, you can get a reliable answer in seconds.

For the "Old Soil" (Oxisol): It works almost perfectly.
For the "Young Soil" (Inceptisol): It works well, though it needs a bit more care.

The study suggests that farmers and consultants can use this "magic flashlight" to make faster decisions about how to feed their crops, saving time and money while being kinder to the environment.

Technical Summary: Spectroscopy Analysis with Machine Learning for Soil Carbon and Nitrogen Quantification

Problem Statement
Traditional soil analysis methods for determining Carbon (C) and Nitrogen (N) content, such as Walkley–Black and Kjeldahl, are time-consuming, labor-intensive, and involve hazardous chemical reagents. While Near-Infrared (NIR) spectroscopy offers a rapid, non-destructive alternative, the transition from benchtop to portable devices introduces challenges regarding spectral resolution, noise, and the need for robust data processing. Furthermore, soil heterogeneity—specifically between different soil classes like Oxisols and Inceptisols—can significantly impact model generalization. This study addresses the need to develop reliable, portable NIR-based predictive models for C and N quantification in these specific Brazilian soil types, optimizing preprocessing and machine learning (ML) strategies to overcome data limitations and soil variability.

Methodology
The study utilized 153 soil samples (72 Oxisol, 81 Inceptisol) collected from various locations. Spectral data were acquired using a portable MyNIR device (900–1700 nm range), while reference C and N values were established via Walkley–Black and Kjeldahl methods, respectively.

The methodological workflow involved:

Data Preprocessing: A systematic evaluation of various techniques was conducted, including Multiplicative Scatter Correction (MSC), Standard Normal Variate (SNV), Logarithmic transformation, Savitzky-Golay (SG) filtering, and adaptive spectral trimming. A critical component was the implementation of a robust outlier removal method combining the Nonlinear Iterative Partial Least Squares (NIPALS) algorithm with a Huber loss function to identify samples with disproportionate prediction errors.
Validation Strategies: The study compared 10-fold cross-validation, Leave-One-Out (LOO), and Holdout methods using the Kennard-Stone algorithm for train/test splitting. The optimal split was determined to be 70/30 for Oxisols and 75/25 for Inceptisols to ensure statistical representativeness.
Modeling: Several regression algorithms were tested, including Partial Least Squares (PLS), Support Vector Regression (SVR), Ridge Regression, and Random Forest (RF). Additionally, Stacking Ensemble Learning models were constructed, using PLS, SVR, and Ridge as base estimators with Linear Regression as the meta-model.
Feature Importance: Ridge Regression coefficients were analyzed to identify the most and least influential wavelengths for prediction.

Key Results

Preprocessing: The combination of the Savitzky-Golay filter and the Huber-based outlier removal method yielded the best performance across all scenarios. Isolated techniques like SNV and MSC were found inadequate for these specific soil types.
Soil Type Performance: A distinct performance gap was observed between soil types. Oxisol models demonstrated superior predictive capability compared to Inceptisol models.
- Oxisols: The SVR model with data standardization (Pipeline 2) achieved the best results for both components. For Carbon, it reached $R^2 = 0.91$ , RMSE = 0.36, and RPD = 3.39. For Nitrogen, it achieved $R^2 = 0.89$ , RMSE = 0.03, and RPD = 3.01. These metrics indicate excellent predictive capability (RPD > 3.0).
- Inceptisols: Results were satisfactory but less expressive. For Carbon, the Stacking Ensemble (Pipeline 6) performed best ( $R^2 = 0.79$ , RPD = 2.16). For Nitrogen, PLS (Pipeline 4) was optimal ( $R^2 = 0.77$ , RPD = 2.11). The lower performance is attributed to the narrower range and lower variability of C and N contents in Inceptisol samples compared to Oxisols.
Model Behavior: Random Forest models consistently exhibited signs of overfitting, showing high calibration $R^2$ but significantly lower validation scores. Stacking ensembles provided more consistent results across calibration and validation sets.
Feature Analysis: The Ridge Regression analysis identified the 1300–1400 nm spectral region as the most critical for predicting both C and N, correlating with C–H and N–H/O–H bond absorptions. Conversely, the 1600–1700 nm and 950–1000 nm regions were identified as the least influential.

Significance and Contributions
The paper claims that portable NIR spectroscopy, when coupled with specific preprocessing (SG + Huber outlier removal) and appropriate ML pipelines, is a viable tool for the rapid quantification of soil C and N. The study highlights that:

Soil Specificity is Crucial: Model performance is heavily influenced by pedological characteristics and the natural variability of the target analytes within the specific soil class.
Optimization of Portable Data: Despite the lower resolution of portable devices compared to benchtop models, robust preprocessing can yield high-quality predictive models (RPD > 2.0 for Inceptisols and > 3.0 for Oxisols).
Practical Application: The developed technique supports faster decision-making for agricultural producers and consultants regarding organic matter content, fertility, and nutrient availability, aligning with the demand for sustainable and efficient analytical methods.
Spectral Insight: The identification of the 1300–1400 nm region as the primary driver for prediction offers a pathway for developing lower-cost, targeted sensors by potentially discarding less relevant spectral bands.

The authors conclude that while the approach is promising, the variability in soil types necessitates tailored modeling strategies, and future work should focus on variable selection algorithms to further simplify these models.