🤖 AI

Illumination-Robust Camera-Based Heart-Rate Estimation for Physiological Sensing in Robots

This paper proposes an illumination-robust, end-to-end spatial-temporal transformer framework for remote heart-rate estimation that integrates 3D face alignment, illumination augmentation, and hybrid temporal-frequency supervision to achieve a 93.6% reduction in mean absolute error compared to the PhysFormer baseline under varying lighting conditions.

Original authors: Zhi Wei Xu, Torbjörn E. M. Nordling

Published 2026-06-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhi Wei Xu, Torbjörn E. M. Nordling

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot that wants to know how you're feeling. Instead of asking you or attaching a sensor to your wrist (which feels awkward and requires you to share private data), the robot just looks at you with its camera. It tries to read your heartbeat by watching the tiny, almost invisible color changes in your face caused by blood pumping through your skin. This technology is called remote photoplethysmography (rPPG).

However, there's a big problem: lighting. Just like trying to hear a whisper in a noisy room, the robot's "heartbeat signal" is very weak. If the room lights change, or if you move from a bright window to a dark corner, the robot gets confused. The change in brightness drowns out the subtle color pulse, making the robot's guess wildly inaccurate.

This paper presents a new "super-ear" for robots that can hear that heartbeat whisper even when the lighting is messy. Here is how they built it, explained simply:

1. The Problem: The "Flickering Spotlight"

Think of the robot's camera as a detective trying to spot a faint red glow on your face. But the room lights keep changing from dim to bright. The detective gets distracted by the changing shadows and stops seeing the glow. Previous methods were like detectives who gave up when the lights flickered.

2. The Solution: A Three-Part Toolkit

The authors built a system with three specific tricks to help the robot ignore the bad lighting and focus on the heartbeat.

Trick A: The "Steady Hand" (Face Alignment)
Before the robot even starts listening, it uses a tool called PRNet. Imagine this as a digital artist who instantly draws a perfect, stable outline of your face, no matter how you tilt your head or how the light hits your nose. This ensures the robot is always looking at the exact same patch of skin, so it doesn't get confused by your face moving around.

Trick B: The "Practice Room" (Illumination Augmentation)
During training, the robot doesn't just learn in one perfect room. The researchers created a "practice room" where they artificially messed up the lighting for every video clip. They made the video suddenly brighter or darker, but they did it to the whole clip at once, not frame-by-frame.

The Analogy: Imagine practicing a song. If you change the volume of every single note randomly, you can't hear the melody. But if you practice the whole song in a loud room, then a quiet room, then a room with a bass boost, you learn to recognize the melody (the heartbeat) regardless of the volume (the lighting).

Trick C: The "Noise-Canceling Headphones" (Residual Temporal Standardization)
This is the paper's most clever invention. Inside the robot's brain, there is a special module called RTSM.

The Analogy: Imagine you are trying to hear a friend's voice in a windy park. The wind (lighting changes) makes the air pressure fluctuate wildly. The RTSM is like a smart filter that says, "Okay, the air pressure is shifting, but let's keep the shape of your voice and just smooth out the wind gusts." It doesn't delete your voice; it just removes the weird bumps caused by the wind, leaving the pure signal behind.

3. The "Double-Check" System (Hybrid Loss)

To teach the robot, the researchers used two different ways to grade its homework:

The Waveform Check: Does the shape of the heartbeat line look right? (Time domain)
The Rhythm Check: Does the beat happen at the right speed? (Frequency domain)

They found that relying on just one of these checks wasn't enough. If they only checked the shape, the robot got confused by the lights. If they only checked the speed, it missed the details.

The Sweet Spot: They tested different "weights" for these two checks. They discovered that when they gave the Rhythm Check a specific amount of importance (a setting they call β = 5), the robot performed perfectly. It was like finding the perfect mix of bass and treble in a music equalizer.

4. The Results: From Clueless to Crystal Clear

The researchers tested their new robot brain against an older, standard model (called PhysFormer) using a dataset where the lighting changed between low, medium, and high levels.

The Old Model: It was almost useless. It had a correlation of only 0.088 (basically guessing randomly) and made huge errors.
The New Model: With their new toolkit, the robot achieved a correlation of 0.982.
- The Analogy: If the old model was like someone trying to guess the time by looking at a cloudy sky, the new model is like looking at a perfect digital clock.
- The error dropped by 93.6%. The robot went from being completely unreliable to being incredibly accurate, even when the lights changed.

Summary

This paper doesn't claim the robot can now diagnose diseases or run a hospital. It simply proves that by using stable face tracking, smart lighting practice, and a special noise-canceling filter, robots can finally read human heart rates through a camera without being thrown off by a change in the room's lighting. It turns a fragile, lighting-sensitive trick into a robust, reliable tool for future human-robot interactions.

Technical Summary: Illumination-Robust Camera-Based Heart-Rate Estimation for Physiological Sensing in Robots

Problem Statement
Remote photoplethysmography (rPPG) offers a non-contact method for estimating heart rate (HR) using standard RGB cameras, making it a promising modality for physiological sensing in service, social, and assistive robots. However, robust deployment in real-world human-robot interaction (HRI) environments is hindered by illumination variation. In dynamic settings, changes in room lighting, viewing angles, and user positions can dominate the weak pulse-related color signals, causing significant performance degradation in existing rPPG models. While prior studies have explored temporal-frequency hybrid supervision, the specific balance required to maintain robustness under controlled illumination variation remains insufficiently examined.

Methodology
The authors propose an end-to-end spatial-temporal transformer framework designed specifically for illumination-robust HR estimation. The system integrates four key components:

Preprocessing: Raw videos undergo 3D face alignment using PRNet to generate semantically aligned facial regions via UV position map regression. This ensures stable facial inputs before feature extraction.
Data Augmentation: To simulate cross-video illumination variation, a clip-level brightness and contrast perturbation is applied. Unlike frame-level randomization, this approach applies consistent $\alpha_c$ (contrast) and $\beta_b$ (brightness) offsets across all frames within a 10-second clip (500 frames at 50 fps), preserving temporal continuity while altering global video appearance.
Architecture: The backbone is a PhysFormer-style temporal-difference Transformer, utilizing Temporal-Difference Multi-Head Self-Attention (TD-MHSA) and Spatio-Temporal Feed-Forward (ST-FF) blocks. A novel Residual Temporal Standardization Module (RTSM) is inserted after the convolutional stem and before tube-token embedding. RTSM computes temporal mean and standard deviation along the temporal dimension for each channel and spatial location, applying a learnable residual scaling parameter ( $\alpha$ ) to correct distribution shifts in intermediate features without erasing the original physiological signal.
Loss Function: The training objective employs a hybrid temporal-frequency loss: $L_{total} = L_{time} + \beta L_{freq}$ $L_{t o t a l} = L_{t im e} + β L_{f r e q}$ .
- $L_{time}$ : A Soft-Shifted Pearson loss that calculates correlation over a shift set ( $S = \pm 3$ frames) to tolerate minor temporal misalignments.
- $L_{freq}$ : A spectral Kullback-Leibler (KL) divergence loss that aligns the power spectrum of the predicted waveform with the ground truth within the 40–180 bpm range.
- $\beta$ : A tunable weight controlling the contribution of frequency-domain guidance.

Key Contributions
The paper makes three primary contributions:

Problem Formulation: It frames rPPG-based HR estimation as a camera-based physiological perception problem specifically for human-facing robotic systems under illumination variation.
Architectural Innovation: It introduces the RTSM, which acts as a residual correction to suppress illumination-induced temporal feature-statistic shifts while preserving subtle pulse patterns.
Empirical Analysis of Hybrid Supervision: It investigates the tuning of temporal-frequency supervision weights on a subject-independent, multi-illumination protocol. The study identifies a practical balance between temporal waveform consistency and spectral HR guidance, demonstrating that neither temporal nor spectral supervision alone is sufficient for robust performance under varying lighting.

Results
Experiments were conducted on a dataset containing 75 subjects under three illumination levels (Low, Medium, High) using a static all-level mix protocol.

Optimal Weighting: Among tested $\beta$ values, $\beta = 5$ yielded the strongest results, achieving a best-run Heart Rate Mean Absolute Error (MAE) of 0.79 bpm and a correlation coefficient of 0.982.
Performance Gains: Compared to the PhysFormer baseline evaluated on the same dataset, the proposed estimator reduced HR MAE by 93.6% (from 12.33 bpm to 0.79 bpm) and increased HR correlation from 0.088 to 0.982.
Component Ablation:
- Using the hybrid loss alone reduced MAE to 3.54 bpm.
- Adding illumination augmentation further reduced MAE to 2.33 bpm.
- Adding RTSM alone reduced MAE to 1.25 bpm.
- The full combination of all components achieved the final 0.79 bpm MAE.
Learnable Parameters: The learnable residual coefficient $\alpha$ in the RTSM converged to negative values, indicating the model learns to use the standardized temporal component as a suppressive correction to reduce non-physiological variation.

Significance and Claims
The paper claims that combining objective-level hybrid supervision, feature-level residual temporal standardization, and data-side illumination augmentation is critical for robust rPPG learning in varying lighting conditions. The authors position this work as a controlled first step toward robust physiological sensing for robots, isolating illumination variation before addressing motion and exercise factors. They assert that the proposed estimator makes camera-based HR estimation usable when illumination varies, a prerequisite for unobtrusive human-aware robot perception. The study does not claim to solve motion robustness or online adaptation, noting these as subjects for future work involving speaking, head-rotation, and bicycle-exercise tasks.