⚡ electrical engineering

Reliability-Aware Weighted Multi-Scale Spatio-Temporal Maps for Heart Rate Monitoring

This paper introduces a Reliability-Aware Weighted Multi-Scale Spatio-Temporal (WMST) map combined with a novel Self-Supervised Learning approach using Swin-Unet and a High-High-High wavelet negative example to significantly improve the robustness and accuracy of remote heart rate monitoring under challenging illumination and motion conditions.

Original authors: Arpan Bairagi, Rakesh Dey, Siladittya Manna, Umapada Pal

Published 2026-03-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Arpan Bairagi, Rakesh Dey, Siladittya Manna, Umapada Pal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to listen to a friend whisper a secret in a crowded, windy room. The "whisper" is your heartbeat, and the "crowded room" is the real world, full of bad lighting, people moving around, and shadows.

This paper introduces a new way to listen to that whisper using just a video camera, without touching the person. Here is the breakdown of their solution using simple analogies:

1. The Problem: The "Static" in the Signal

Remote Photoplethysmography (rPPG) is a fancy term for measuring your heart rate by watching tiny color changes in your skin (caused by blood pumping) using a camera.

The Issue: In a perfect studio, this is easy. But in the real world, the camera gets confused. Sunlight glares, shadows move, and if you turn your head, the camera thinks the movement is your heartbeat. It's like trying to hear a whisper while someone is banging on a drum next to you.

2. The Solution: The "Smart Filter" (WMST Map)

The authors created a new tool called the Reliability-Aware Weighted Multi-Scale Spatio-Temporal (WMST) Map.

The Analogy: Imagine you are a detective looking at a crime scene photo. Some parts of the photo are blurry or covered in fog (bad lighting), and some parts are crystal clear.
How it works: Instead of looking at the whole face equally, their algorithm acts like a smart detective. It puts a "trust score" on every pixel of the video:
- High Trust: Smooth skin on your cheek (where the blood flow is visible).
- Low Trust: Shiny forehead (glare), hair (no skin), or moving shadows.
The Result: The system automatically turns down the volume on the "noisy" parts of the video and turns up the volume on the "reliable" parts. It's like putting on noise-canceling headphones that only let the heartbeat through.

3. The Training Method: "Teaching by Contrast"

To teach the computer how to find the heartbeat without needing a doctor to label every video (which is hard and expensive), they used a technique called Self-Supervised Learning.

The Analogy: Imagine you are teaching a child to recognize a "real apple."
- The Positive Example: You show them a real apple.
- The Negative Example: You show them a fake plastic apple.
- The Trick: Most previous methods showed the child a fake apple that looked nothing like a real one (like a banana). The child learned too easily.
The Innovation (The HHH Map): The authors created a special "fake apple" called the HHH Wavelet Map.
- This fake apple looks exactly like the real one in terms of shape and movement (so the child can't just say "it's moving, so it's fake"), but it has zero apple flavor (no heartbeat signal).
- By forcing the computer to tell the difference between a "heartbeat video" and this "motion-only video," the computer learns to ignore movement and focus only on the subtle color changes of the blood.

4. The Result: A Clearer Whisper

They tested this on two big datasets (one with people moving around in bad light, one in a calm room).

The Outcome: Their method was better at ignoring the "wind and noise" than any previous method.
The Stats: It made fewer mistakes in guessing the heart rate and matched the "real" heart rate more closely.

Summary

Think of this paper as inventing a super-smart pair of glasses for a camera.

The Glasses automatically blur out the glare, shadows, and hair (the WMST Map).
The Training teaches the camera to ignore the motion of the head and focus only on the blood flow, using a tricky "fake heartbeat" video as a test (the HHH Map).

The result? A camera that can accurately tell your heart rate even if you are walking outside on a sunny, windy day.

1. Problem Statement

Remote Photoplethysmography (rPPG) estimates physiological signals (like Heart Rate) from facial videos by analyzing subtle skin color changes caused by blood volume pulses. However, in unconstrained environments, rPPG signals suffer from low Signal-to-Noise Ratio (SNR) due to:

Environmental Noises: Illumination changes, shadows, and specular reflections.
Motion Artifacts: Head movements and facial expressions.
Data Limitations: Deep learning methods typically require large-scale annotated physiological data, which is difficult to acquire.
Existing SSL Limitations: Current Self-Supervised Learning (SSL) approaches often use generic negative augmentations that are either too easily separable or fail to preserve realistic facial dynamics, and they rely on standard Multi-Scale Spatio-Temporal (MST) maps that do not adequately model or suppress noise.

2. Methodology

The authors propose a framework combining advanced pre-processing with a contrastive learning strategy based on Swin-Unet. The core components are:

A. Reliability-Aware Weighted Multi-Scale Spatio-Temporal (WMST) Map

Instead of simple mean pooling used in traditional MST maps, the WMST map assigns adaptive weights to pixels based on their reliability for physiological signal extraction.

Skin Probability: Uses chrominance (U, V channels) and Mahalanobis distance to weight skin regions higher.
Edge/Motion Suppression: Uses Stationary Wavelet Transformation (SWT) to detect and suppress high-frequency edge responses (HL, LH, HH subbands) that indicate motion boundaries or non-physiological textures.
Reflection Handling: Uses HSV color space to identify specular highlights (high Value, low Saturation) and suppresses them via an exponential weighting function.
Shadow Handling: Penalizes dark pixels (low Luminance Y) caused by shadows.
Adaptive Fusion: These individual weights are aggregated using variance-based weighting (Equation 6) to create a final reliability map, which is then used for spatial ROI-wise weighted mean pooling.

B. Contrastive Learning Framework

The model learns robust representations without ground-truth labels using a contrastive approach:

Anchor: The generated WMST map.
Positive Pairs: Generated by concatenating 1D physiological signals derived from traditional methods (ICA, PCA, CHROM, etc.) and temporally resized/augmented WMST maps.
Negative Pairs:
1. WMST maps from different videos.
2. High-High-High (HHH) Wavelet Map: A novel negative sample created by applying 3D SWT to the input video and extracting the HHH subband. This map retains motion and structural details (high-frequency variations) but filters out physiological information, making it a "difficult" negative that forces the model to distinguish between motion noise and true physiological signals.
Loss Function: A temperature-free InfoNCE loss using the Pearson correlation coefficient on the Power Spectral Density (PSD) within the [0.5, 3.0] Hz band.

C. Network Architecture & Training

Backbone: Swin-Unet, chosen for its ability to model long-range temporal dependencies via self-attention.
Task Formulation: The model is trained to reconstruct a refined spatio-temporal map from the input WMST map, rather than directly regressing Heart Rate.
Two-Stage Training:
1. Pre-training: Trained on signals from traditional methods to learn physiologically relevant weights (minimizing L1 distance and maximizing correlation).
2. Contrastive Learning: Fine-tuned using the proposed positive/negative pairs to enhance robustness.
Inference: The output map is averaged and band-pass filtered; HR is estimated from the dominant frequency of the PSD.

3. Key Contributions

WMST Map: A novel pre-processing technique that models pixel reliability by suppressing environmental noises (reflection, shadow, motion) using statistical and mathematical approaches, resulting in significantly higher SNR than standard MST maps.
HHH Wavelet Map: The introduction of a specific negative sample (HHH map) for contrastive learning that explicitly models motion and structural noise while removing physiological signals, improving the model's ability to filter out artifacts.
SSL Framework: A complete self-supervised pipeline that achieves state-of-the-art performance without relying on large-scale annotated physiological ground truth.
Deterministic Efficiency: The weighting process is non-iterative and offline, maintaining computational complexity similar to standard MST maps ( $O(H \times W \times T)$ ).

4. Experimental Results

The method was evaluated on two public benchmarks: VIPL-HR (unconstrained, high motion/noise) and UBFC-rPPG (controlled, stable).

Statistical Significance: One-sided t-tests and Wilcoxon signed-rank tests confirmed that WMST significantly improves SNR over traditional MST maps ( $p < 10^{-13}$ ).
Quantitative Performance (VIPL-HR):
- MAE: 4.32 (Best among SSL methods, outperforming RS+rPPG's 4.40).
- Pearson Correlation (R): 0.86 (Highest among all compared SSL methods).
- RMSE: 7.26 (Competitive, slightly higher than RS+rPPG's 7.19, but with better correlation).
Quantitative Performance (UBFC-rPPG):
- Achieved an MAE of 0.50 and R of 0.99, comparable to other SOTA SSL methods, demonstrating generalizability to controlled settings.
Ablation Study:
- Removing the WMST map increased MAE from 4.32 to 4.55.
- Removing the HHH map decreased Pearson correlation from 0.86 to 0.80, proving the HHH map's role in signal synchronization.
Qualitative Analysis: Visual comparisons showed the proposed method tracks ground truth more consistently, with fewer spurious peaks and overshoots compared to competitors like RS+rPPG, especially in segments with double-peak patterns or sharp transitions.

5. Significance

This paper addresses a critical bottleneck in remote physiological monitoring: the degradation of signal quality in real-world, unconstrained environments. By shifting from generic data augmentation to reliability-aware signal modeling (WMST) and physics-informed negative sampling (HHH map), the authors demonstrate that SSL can achieve superior robustness without labeled data. The approach offers a practical, computationally efficient solution for deploying rPPG in mobile and real-time applications where lighting and motion cannot be controlled.