Dynamic Sparse Partial Least Squares for Sensor Fault Detection and Diagnosis in Smart Agriculture

Problem Statement

Reliable sensor measurements are critical for precision agriculture, where faulty sensors can lead to inaccurate agronomic decisions, resource inefficiency, and reduced crop yields. While smart agriculture increasingly relies on IoT platforms to monitor soil and environmental conditions, existing fault detection methods face significant challenges:

• Data Scarcity: Supervised learning methods require labeled fault data, which is often unavailable because sensor faults are infrequent, diverse, and difficult to annotate.
• Complex Dynamics: Agricultural measurements exhibit strong temporal dependencies, seasonal variability, and non-Gaussian behaviors due to weather, irrigation, and soil physics. Static monitoring methods often fail to capture these dynamic interactions.
• Interpretability and Redundancy: Conventional multivariate methods (e.g., PCA, static PLS) often retain all variables, including redundant or noisy ones, and ignore time-lagged information, reducing detection sensitivity and interpretability.
• Distributional Assumptions: Many statistical monitoring techniques assume Gaussian distributions, which may not hold for agricultural data, leading to unreliable false-alarm rates.

Methodology: The D-sPLS Framework

The authors propose a Dynamic Sparse Partial Least Squares (D-sPLS) framework designed for semi-supervised sensor fault detection and diagnosis. The framework operates exclusively on fault-free (anomaly-free) historical data, eliminating the need for labeled fault samples. It consists of four main stages:

1. Dynamic Feature Construction:

   • To capture temporal dependencies, the current measurement vector $x_t$ is augmented with historical observations ($x_{t-1}, \dots, x_{t-L}$) to form a dynamic vector $\tilde{x}_t$.
   • This allows the model to learn delayed process responses (e.g., the effect of rainfall on soil moisture) rather than relying solely on instantaneous correlations.

2. Sparse Latent-Variable Modeling:

   • The framework employs a Sparse PLS (sPLS) formulation to project the high-dimensional, dynamically augmented predictor matrix onto a compact latent space.
   • An $\ell_1$ penalty is applied to the predictor weights to enforce sparsity. This performs embedded feature selection, retaining only the most informative sensors and time lags while suppressing redundant variables and noise.
   • The model maximizes the covariance between the selected predictors and the target sensor, creating a predictive model of normal operating behavior.

3. Residual-Based Monitoring (EWMA + KDE):

   • Prediction: The D-sPLS model predicts the expected value of the target sensor. The difference between the measured and predicted values generates a residual sequence.
   • Smoothing: To enhance sensitivity to small, persistent faults (like drift) while attenuating random noise, the residuals are transformed into an Exponentially Weighted Moving Average (EWMA) statistic.
   • Thresholding: Instead of assuming a Gaussian distribution, the decision threshold is estimated using Kernel Density Estimation (KDE) on the fault-free training residuals. This nonparametric approach adapts to skewed or multimodal data distributions, ensuring robust false-alarm control.

4. Diagnostic Layer (ICE-Based):

   • Upon fault detection, an Individual Conditional Expectation (ICE) layer is activated to provide interpretability.
   • ICE curves visualize the relationship between a specific predictor and the target sensor while holding other variables constant.
   • By comparing actual measurements against the learned ICE envelope, operators can distinguish between abrupt bias faults (sudden displacement) and gradual drift (progressive departure) and assess fault severity.

Key Contributions

1. Semi-Supervised Dynamic Sparse Modeling: The integration of temporal feature augmentation with sparse latent-variable modeling allows the system to capture time-lagged interactions and select informative predictors without requiring fault labels.
2. Robust Nonparametric Monitoring: The combination of EWMA statistics with KDE-based thresholds provides a distribution-free monitoring strategy suitable for the non-Gaussian and nonstationary nature of agricultural data.
3. Interpretability: The inclusion of an ICE-based diagnostic layer moves beyond binary alarms, offering visual insights into fault origin and severity.
4. Comprehensive Validation: The framework is rigorously tested on two real-world agricultural datasets (smart soil monitoring in Ecuador and soil quality assessment in Greece) under various bias and drift fault scenarios, with statistical significance confirmed via bootstrap confidence intervals and Wilcoxon signed-rank tests.

Experimental Results

The proposed D-sPLS framework was compared against a wide range of benchmarks, including static PLS, PCA, ElasticNet, Decision Trees, Support Vector Regression (SVR), Gaussian Process Regression (GPR), and Deep Learning methods (ELM).

• Fault Detection Performance:

  • Bias Faults: D-sPLS consistently achieved the highest True Positive Rates (TPR) and F1-scores across all bias levels (10% to 50%). Notably, it maintained zero false alarms even at low bias levels (10%), whereas other high-performing methods (like GPR and PCA-SPE) exhibited false alarm rates of 2–4%.
  • Drift Faults: For gradual drift faults (5% to 20%), D-sPLS demonstrated superior sensitivity, achieving high detection rates with zero false alarms. It outperformed static models which struggled to detect early-stage drift.
  • Statistical Significance: Pairwise Wilcoxon signed-rank tests confirmed that D-sPLS significantly outperformed all competing methods in terms of F1-score, with large effect sizes.

• Computational Efficiency:

  • Despite the added complexity of dynamic feature construction and sparse modeling, the online execution time per monitoring cycle was approximately 0.30 ms. This is well within the limits for real-time deployment on standard hardware (Intel Core i9, 16GB RAM), confirming the method's suitability for edge IoT applications.

• Diagnosis:

  • The ICE-based analysis successfully visualized the progression of faults. For bias faults, measurements shifted abruptly outside the normal envelope; for drift faults, a gradual migration outside the envelope was observed, allowing for clear differentiation and severity assessment.

Significance and Claims

The paper claims that the proposed D-sPLS framework offers a promising solution for accurate, robust, and interpretable sensor fault detection in smart agricultural systems. Its significance lies in:

• Practicality: By relying solely on normal operating data, it addresses the critical bottleneck of scarce labeled fault data in real-world agriculture.
• Adaptability: The dynamic and sparse nature of the model makes it specifically suited for the coupled, time-dependent, and noisy environment of soil monitoring.
• Reliability: The nonparametric thresholding ensures reliable operation regardless of the underlying data distribution, a common limitation in agricultural sensing.
• Actionability: The diagnostic layer transforms abstract statistical alarms into actionable insights for maintenance and recalibration.

The authors conclude that while the current work addresses single-sensor faults, the framework provides a flexible foundation for future research into adaptive online updating, multi-fault isolation, and integration with edge-enabled IoT systems.