💻 computer science

Decision-Level Fusion for Robust Wearable Affect Recognition

This paper proposes a robust affect recognition framework for wearable devices that combines Fourier-Bessel Series Expansion with Empirical Wavelet Transform for transient feature extraction and employs uncertainty-weighted decision-level fusion, demonstrating superior performance over feature-level aggregation on the WESAD dataset under heterogeneous and noisy sensing conditions.

Original authors: Lokesh Singh, Athina Georgara, Jayati Deshmukh, Tan Viet Tuyen Nguyen, Sarvapali D. Ramchurn

Published 2026-05-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lokesh Singh, Athina Georgara, Jayati Deshmukh, Tan Viet Tuyen Nguyen, Sarvapali D. Ramchurn

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Reading the Mind's Weather

Imagine you want to know if someone is stressed, happy, or just chilling out. Instead of asking them, you look at their body's "weather report." This is done using wearable gadgets (like smartwatches) that measure things like heart rate, skin sweat, and muscle tension.

The problem is that body signals are messy. They change quickly, they get noisy (like static on a radio), and sometimes a sensor might stop working or get covered by a sleeve. The authors of this paper built a new system to read these messy signals more accurately and reliably.

The Problem: The "Blurry Camera" Approach

Most current systems look at body signals like a camera with a slow shutter speed. They take a chunk of time, average everything out, and produce a blurry picture.

The Issue: Emotions often happen in quick, sharp bursts (like a sudden spike in heart rate when you get scared). If you "blur" the data to make it smooth, you lose those important, fast details.
The Result: The system might miss the difference between "excited" and "stressed" because it smoothed out the clues.

The Solution: A Three-Step Detective Team

The authors propose a new method that works in three stages, like a team of detectives solving a case.

Step 1: The "High-Res Lens" (Feature Extraction)

Instead of taking a blurry average, the team uses a special mathematical tool called FBSE-EWT.

The Analogy: Imagine listening to a song. A standard method might just tell you the "average volume." This new method acts like a high-quality audio engineer who can separate the song into individual instruments (drums, bass, vocals) even if they are playing at the same time.
What it does: It breaks the body signals down into their specific, fast-moving parts without losing the details. It captures the "transient" (short-lived) spikes that actually tell us how a person feels.

Step 2: The "Specialist Detectives" (Single-Sensor Prediction)

Once the signals are clear, the system doesn't just throw them all into one big blender. Instead, it assigns a different "detective" to each sensor.

The Setup: One detective looks only at the heart (ECG), another only at the skin sweat (EDA), and another only at movement (Accelerometer).
The Job: Each detective makes their own guess: "I think this person is stressed."
The Confidence Check: Crucially, each detective also rates their own confidence. "I'm 90% sure," or "I'm only 40% sure because my signal is noisy."

Step 3: The "Wise Council" (Decision-Level Fusion)

This is the most important part. Instead of combining the raw data (which is messy), the system combines the decisions of the detectives.

The Analogy: Imagine a jury. If one juror is shouting loudly but is clearly confused (low confidence), the others ignore them. If another juror is calm, sure of their facts, and has a clear view (high confidence), their vote counts for more.
How it works: The system takes the guesses from all the sensors and weighs them.
- If a sensor is noisy or broken, the system gives its vote very little weight.
- If a sensor is clear and confident, its vote counts heavily.
The Benefit: If one sensor fails completely (like a watch falling off), the system doesn't crash. It just listens to the remaining detectives and makes a decision based on who is still reliable.

The Results: Does It Work?

The team tested this on a standard dataset called WESAD (which includes 15 people wearing sensors while they were relaxed, stressed, or amused).

The Comparison: They compared their "Wise Council" method against the old "Blender" method (feature-level fusion).
The Outcome:
- In about 84% of the cases, the new method was just as good or better than the old one.
- In about 48% of the cases, the new method was strictly better.
The Takeaway: The new method is more robust. It handles missing sensors and noisy data much better than the traditional approach.

Summary

Think of this paper as upgrading from a group of people shouting random facts into a room to a well-organized committee.

They use better tools to hear the facts clearly (FBSE-EWT).
They let each expert speak for themselves (Single-Sensor Predictors).
They vote based on who is most reliable at that exact moment (Uncertainty-Weighted Fusion).

This makes the system much better at figuring out how you feel, even when your watch is slipping off or your heart rate is jumping around.

Technical Summary: Decision-Level Fusion for Robust Wearable Affect Recognition

Problem Statement
Automatic recognition of affective states (specifically baseline, stress, and amusement) from wearable physiological signals faces significant challenges in real-world deployment. While datasets like WESAD facilitate evaluation, current pipelines struggle with the non-stationary, noisy, and subject-dependent nature of physiological time series. Two primary limitations are identified:

Feature Extraction: Common methods relying on fixed-basis spectral features (e.g., FFT bandpower, Welch PSD) assume local stationarity. These approaches often oversmooth short-lived, discriminative transient patterns critical for distinguishing affective states.
Multimodal Integration: Real-world sensing is prone to artifacts, signal failures, and missing modalities. Traditional early fusion (feature-level) approaches often assume complete sensor availability, leading to performance degradation when sensors are noisy or unavailable.

Methodology
The authors propose a three-stage pipeline designed to address non-stationarity and sensor unreliability:

Non-Stationary Feature Extraction (FBSE-EWT):
Instead of fixed-window Fourier transforms, the paper introduces a hybrid signal analysis method combining Fourier–Bessel Series Expansion (FBSE) with Empirical Wavelet Transform (EWT).
- FBSE: Represents the windowed signal using Bessel bases, which exhibit amplitude-modulation-like behavior suitable for wide-band, non-stationary signals.
- EWT: Performs data-driven spectral segmentation. Unlike fixed wavelet banks, EWT learns wavelet bands directly from the signal's spectrum by detecting meaningful minima in a histogram of the FBSE spectrum.
- Feature Construction: The method reconstructs modes from these adaptive filters and extracts fixed-dimensional descriptors, including mode energy, log-energy, and mode entropy. This preserves transient spectral structures that fixed-basis methods might discard.
Single-Sensor Prediction (SSP):
Each modality (ECG, EDA, BVP, EMG, ACC) is processed independently by a dedicated predictor. The authors utilize a Multi-Layer Perceptron (MLP) with ReLU activation and a softmax output layer.
- The MLP outputs a probability distribution ( $P_i$ ) over the affective classes and a confidence score ( $F1_i$ ) representing the predictor's reliability.
- This modular design allows each sensor to learn affect-discriminative patterns independently, avoiding the complexity of temporal models (like CNNs/LSTMs) while maintaining efficiency for smaller datasets.
Uncertainty-Aware Decision-Level Aggregation:
Rather than fusing raw data or features, the pipeline aggregates the independent predictions at the decision level. The final prediction is a weighted average of individual sensor probabilities, where weights are determined by:
- Predictive Uncertainty: Measured via the information entropy ( $H(P_i)$ ) of the specific prediction instance. Lower entropy (higher certainty) increases the weight.
- Modality Reliability: Measured by the global F1 score ( $F1_i$ ) of the single-sensor predictor.
- Fusion Rule: The aggregated probability for a class $e$ is calculated as:
  $P_T(e) = \frac{1}{\gamma} \sum_{i \in T} (1 - H(P_i))^{F1_i} \cdot P_i(e)$
  where $\gamma$ is a normalization factor. This design ensures that uncertain or unreliable sensors contribute less to the final decision, enabling graceful degradation when sensors fail or are noisy.

Key Contributions

Adaptive Representation: A non-stationary representation (FBSE-EWT) that isolates transient spectral modes via data-driven segmentation, better preserving the dynamic structure of physiological signals than fixed-basis methods.
Robust Fusion Mechanism: A modular, uncertainty-aware decision fusion rule that weights sensor contributions based on both instance-specific entropy and global modality reliability, ensuring stability under missing or noisy data.
Empirical Validation: An evaluation on the WESAD dataset demonstrating the efficacy of this approach for three-class affect recognition.

Results
Evaluated on the WESAD dataset (15 subjects, 5 sensors, 3 classes), the proposed framework yielded the following findings:

Decision vs. Feature Fusion: Decision-level aggregation was found to be ≈84% as good as or better than feature-level aggregation. In ≈48% of cases, decision-level fusion was strictly superior.
Team Size Robustness: Decision-level aggregation consistently outperformed feature-level aggregation across all team sizes (from single sensors to full quartets).
Scalability: Accuracy improved as more sensors participated in the aggregation, confirming the benefit of multimodal integration even when individual sensors vary in quality.

Significance and Claims
The paper claims that the proposed pipeline offers improved robustness for wearable affect recognition in heterogeneous and partially reliable sensing environments. By shifting from feature-level to decision-level fusion and employing adaptive spectral decomposition, the system outputs calibrated belief states rather than overconfident predictions. This is particularly significant for agent-based decision support in health and care settings, where interventions must be triggered conservatively under uncertainty. The authors position this work as a step toward deployable, affect-aware health systems that can handle the realities of non-stationary physiology and sensor failure without requiring expert knowledge of specific signal features for integration.