💻 computer science

HOT: Harmonic-Constrained Optimal Transport for Remote Photoplethysmography Domain Adaptation

This paper introduces HOT, a novel framework that combines Frequency Domain Adaptation (FDA) to transfer appearance-related spectral components and Harmonic-Constrained Optimal Transport to enforce physiological consistency, thereby significantly enhancing the robustness and generalization of remote photoplethysmography (rPPG) models across diverse domains.

Original authors: Ba-Thinh Nguyen, Thi-Duyen Ngo, Thanh-Trung Huynh, Thanh-Ha Le, Huy-Hieu Pham

Published 2026-04-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ba-Thinh Nguyen, Thi-Duyen Ngo, Thanh-Trung Huynh, Thanh-Ha Le, Huy-Hieu Pham

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to measure your heart rate just by looking at a video of your face. This technology is called remote photoplethysmography (rPPG). It works by spotting tiny, almost invisible color changes in your skin caused by your blood pumping.

However, there's a big problem: The robot gets confused when the environment changes.

If you train the robot in a bright studio with a high-end camera, it might fail miserably when you take it outside on a cloudy day with a cheap phone camera. The lighting, the camera quality, and even your skin tone look different. The robot mistakes these "appearance" changes for changes in your heart rate, leading to bad readings.

This paper introduces a clever solution called HOT (Harmonic-Constrained Optimal Transport) to fix this. Here is how it works, explained with simple analogies:

1. The Problem: The "Chameleon" Effect

Think of the robot as a student who memorized a textbook perfectly. But the textbook was written in a specific font, on specific paper, with specific lighting. When the student takes a test written in a different font on a different screen, they get confused.

In rPPG, the "textbook" is the training data. The "font" is the lighting and camera style. The "heart rate" is the actual story the student needs to learn. Current robots learn the font too well and forget the story.

2. The First Trick: "Style Transfer" (Frequency Domain Adaptation)

The authors first try to teach the robot to ignore the "font." They use a technique called Frequency Domain Adaptation (FDA).

The Analogy: Imagine you have a photo of a person (Source) and a photo of the same person in a different room (Target).
The Magic: The computer separates the photo into two parts:
1. The Shape (Phase): The actual structure of the face and the subtle color pulses of the heartbeat.
2. The Lighting/Color (Low-Frequency Amplitude): The brightness, shadows, and color cast of the room.
The Swap: The computer takes the "Shape" from the Source photo and swaps in the "Lighting" from the Target photo.
The Result: The robot now sees a video that looks like it was filmed in the target environment (e.g., outdoors), but the heartbeat signal inside it is still the original, clean one. This forces the robot to learn: "Hey, the lighting changed, but the heartbeat pattern stayed the same. I need to focus on the pattern, not the lighting!"

3. The Second Trick: The "Harmonic Anchor" (HOT)

There's a catch. If you just swap the lighting, the robot might get too confused. It might align the wrong parts of the video (e.g., matching a blink in the source to a blink in the target) just because they look similar, even if the heartbeats don't match up in time.

To fix this, they introduce HOT (Harmonic-Constrained Optimal Transport).

The Analogy: Imagine you are trying to match two different dance routines (Source and Target) to see if they are the same dance.
- Standard Matching: You just look at the dancers' outfits. If they are wearing similar shirts, you match them. This is risky because outfits change, but the dance steps don't.
- The HOT Approach: You know that a heartbeat is like a rhythmic drumbeat. It has a main beat (the heart rate) and a secondary beat (the echo of the pulse). This is the "Harmonic" property.
The Solution: HOT acts like a strict music conductor. It says: "You can match the dancers, but ONLY if their drumbeats are in sync."
It uses a mathematical tool (Optimal Transport) to find the best way to line up the video frames, but it adds a penalty if the rhythm doesn't match. If the robot tries to match a frame where the heart is beating fast with a frame where it's beating slow, the "Harmonic Constraint" slaps its hand and says, "No! That's not the right match!"

4. The Result: A Super-Resilient Robot

By combining these two tricks:

FDA teaches the robot to ignore the "noise" of different cameras and lights.
HOT forces the robot to align the data based on the "rhythm" of the heart, not just the look of the face.

The paper tested this on many different datasets (different cameras, different people, different lights). The result? The robot became much better at measuring heart rates in new, unseen environments. It didn't just memorize the training data; it learned the physics of the heartbeat.

Summary

Think of HOT as a teacher who:

Gives the student practice tests in different fonts and lighting (FDA) so they don't get distracted by the look of the paper.
Checks their answers against a metronome (Harmonic Constraint) to ensure they are actually counting the beats correctly, not just guessing based on the picture.

This makes the technology robust enough to work in the real world, whether you are in a hospital, a gym, or your living room.

1. Problem Statement

Remote Photoplethysmography (rPPG) is a non-contact technique for estimating physiological signals (e.g., heart rate, blood volume pulse) from facial videos. While deep learning-based rPPG methods have achieved state-of-the-art performance on specific datasets, they suffer from significant performance degradation under domain shift.

The Core Issue: Models trained on a source domain (e.g., controlled lighting, specific camera) fail when deployed on a target domain with different conditions (e.g., varying illumination, skin tones, camera characteristics).
Current Limitations: Existing methods often overfit to appearance-related factors (illumination, color response) rather than learning invariant physiological features. Furthermore, while Frequency Domain Adaptation (FDA) is successful in general computer vision, it has not been effectively applied to rPPG because naive appearance transformations can corrupt the extremely subtle color variations that encode cardiac signals.
The Gap: There is a lack of methods that explicitly align source and target data while enforcing physiological consistency (i.e., ensuring the transformed data still represents valid cardiac dynamics).

2. Methodology: The HOT Framework

The authors propose HOT (Harmonic-Constrained Optimal Transport), a framework that combines Frequency-Domain Adaptation (FDA) with a novel Harmonic-Constrained Optimal Transport loss.

A. Frequency-Domain Adaptation (FDA)

FDA is used to generate "style-transferred" training samples that mimic the target domain's appearance while preserving the source domain's physiological content.

Mechanism: It operates in the Fourier domain. For a source video frame $x_t$ $x_{t}$ and a target reference frame $x_r$ $x_{r}$ :
1. Compute the 2D Discrete Fourier Transform (DFT) for both.
2. Extract the Amplitude ( $A$ ) and Phase ( $\Phi$ ) spectra.
3. Low-Frequency Replacement: A low-frequency mask ( $\Omega_\beta$ ) is defined. The low-frequency amplitude components of the source frame are replaced with those from the target reference frame. High-frequency components (which contain fine details and physiological dynamics) are preserved from the source.
4. Reconstruction: The adapted amplitude is combined with the original source phase, and the Inverse DFT is applied to generate the stylized frame $\tilde{x}_t$ .
Rationale: Domain shifts in rPPG are largely driven by camera spectral response and illumination, which manifest in low-frequency amplitudes. By swapping these, the model learns to ignore appearance variations while retaining the phase spectrum, which encodes the subtle color changes caused by blood flow.

B. Cyclic Local Harmonic Descriptor

To ensure the adapted data remains physiologically valid, the method extracts a harmonic descriptor from the temporal features.

Tokenization: The backbone network extracts spatio-temporal features, which are aggregated into a sequence of temporal tokens $Z$ .
Local Cyclic Signal: For each token, a cyclic temporal neighborhood of fixed length $W$ is constructed to capture local dynamics.
Harmonic Ratio: A windowed DFT is applied to this local signal to compute spectral energy. The method identifies the dominant frequency ( $k_1$ , corresponding to the heart rate) and its second harmonic ( $k_2$ ).
Descriptor: The harmonic ratio $r_t = E(k_2) / E(k_1)$ is calculated. This ratio serves as a physiological constraint, as cardiac signals inherently possess strong harmonic structures.

C. Harmonic-Constrained Optimal Transport (HOT)

The core innovation is using Optimal Transport (OT) to align the original source tokens ( $Z$ ) and the FDA-transferred tokens ( $\tilde{Z}$ ) while respecting physiological constraints.

Transport Cost: The cost $C_{ij}$ between a source token $z_i$ and a transferred token $\tilde{z}_j$ is defined as:
$C_{ij} = (1 - \cos(z_i, \tilde{z}_j)) + \lambda_h |r_i - \tilde{r}_j|$
Where the first term measures feature similarity (geometric) and the second term penalizes differences in the harmonic ratio (physiological).
Optimization: The alignment is solved using the Sinkhorn algorithm (entropy-regularized OT) to find the optimal transport plan $P$ that minimizes the total cost.
Training Objective: The final loss combines the standard supervised regression loss (Negative Pearson Correlation) with the HOT alignment loss:
$L = L_{sup} + \gamma L_{HOT}$
This forces the model to produce representations that are not only accurate but also harmonically consistent across appearance-shifted views.

3. Key Contributions

First FDA Application in rPPG: The paper introduces Frequency-Domain Adaptation specifically for rPPG, demonstrating that low-frequency amplitude swapping can effectively simulate cross-domain appearance variations without destroying physiological signals.
Harmonic-Constrained Optimal Transport (HOT): A novel alignment strategy that leverages the harmonic nature of cardiac signals to guide the transport of features between original and adapted samples, ensuring physiological coherence.
Backbone-Agnostic Framework: The method is designed to be a plug-and-play module that improves generalization across various architectures (CNNs and Transformers).

4. Experimental Results

The framework was evaluated on three major datasets: PURE, UBFC-rPPG, and MMPD (a large-scale, multi-domain dataset with significant domain shifts).

Cross-Dataset Performance: Extensive experiments (e.g., training on PURE, testing on MMPD) showed that adding HOT consistently outperformed baseline models (DeepPhys, PhysNet, TSCAN, PhysFormer, etc.).
- Metrics: Significant reductions in Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Mean Absolute Percentage Error (MAPE), alongside higher Pearson correlation coefficients ( $r$ ).
- Robustness: The improvements were most pronounced in the most challenging scenarios (e.g., UBFC-rPPG $\to$ MMPD), where domain shifts are severe due to lighting and motion variations.
Ablation Studies:
- Low-Frequency Ratio ( $\beta$ ): Optimal performance was found at $\beta=0.05$ , balancing domain adaptation with signal preservation.
- Harmonic Constraint ( $\lambda_h$ ): The harmonic penalty was crucial; without it, the alignment was geometrically consistent but physiologically inconsistent.
- Efficiency: HOT adds a computational overhead during training (~11% increase in time, ~98% increase in peak memory due to Sinkhorn iterations) but has negligible impact on inference latency since the alignment is only used during training.

5. Significance and Conclusion

This work addresses a critical bottleneck in the real-world deployment of rPPG: domain generalization. By explicitly modeling appearance variation through FDA and enforcing physiological consistency via HOT, the authors provide a robust solution that allows rPPG models to generalize across diverse cameras, lighting conditions, and subjects.

Impact: The proposed method moves rPPG closer to practical, unobtrusive healthcare monitoring in unconstrained environments.
Limitations & Future Work: The current approach requires unlabeled target-domain reference samples for FDA. Future work aims to reduce this dependency and further optimize the computational cost of the Sinkhorn alignment.

In summary, HOT represents a significant step forward in making rPPG robust to real-world variability by bridging the gap between data-driven appearance adaptation and physiological signal integrity.