🤖 machine learning

CanonicalPhys: Pose-Robust Remote Photoplethysmography via Canonical-Space Priors

CanonicalPhys addresses the sharp performance degradation of deep remote photoplethysmography under varying head poses by introducing a differentiable homography to map facial regions to a canonical frame, thereby enabling the effective application of anatomical priors like the dichromatic reflection model and pulse-phase invariance to significantly reduce heart-rate estimation errors across diverse pose conditions.

Original authors: Hui Wei, Seyedata Jodeiri Seyedian, Xiaobai Li, Guoying Zhao

Published 2026-07-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hui Wei, Seyedata Jodeiri Seyedian, Xiaobai Li, Guoying Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to listen to a friend's heartbeat by holding a stethoscope to their chest. Easy, right? But now, imagine you have to do it using only a video camera, without ever touching them. This is the magic of remote photoplethysmography (rPPG). It's a fancy way of saying "reading your pulse from a video." When your heart beats, it pumps a tiny wave of blood through your face, changing the color of your skin just a fraction of a percent. Cameras are so sensitive now that they can spot these tiny color shifts and turn them into a heart rate. This technology is a big deal because it could let doctors check patients' hearts from miles away, help monitor newborns without sticking needles in them, or even keep an eye on tired drivers.

However, there's a catch. These cameras are great when you are sitting still and looking straight at the lens. But the moment you turn your head, the system gets confused. Think of it like trying to read a map where the streets keep moving around. If you turn your head, the spot on the screen that was your forehead might suddenly be your cheek, or your temple. The computer doesn't know that the "pixel" it's looking at has moved to a different part of your body, so it tries to find a heartbeat in the wrong place, and the reading goes haywire. Scientists have been trying to fix this by feeding the computer more and more examples of people turning their heads, hoping it will eventually "learn" the trick. But what if the problem isn't that the computer isn't smart enough, but that it's looking at the world from the wrong angle?

This is exactly the puzzle tackled by a new method called CanonicalPhys. The researchers argue that head movement isn't just a data problem; it's a coordinate problem. They realized that when you turn your head, the pixels on the screen scramble, but your face itself doesn't change. So, instead of teaching the computer to guess where the pixels are going, they built a clever "digital straightener" that snaps the face back into a perfect, forward-facing position before the computer even tries to read the heartbeat.

Here's how they did it: Imagine you have a photo of a face that is turned to the side. The researchers use a mathematical trick called a homography (think of it as a digital warp tool) to grab four key points on the face—the corners of the eyes and the corners of the mouth—and drag them into fixed, perfect positions, as if the person were looking straight at the camera. Once the face is "straightened" into this canonical (or standard) frame, the pixels stay in the right place. The forehead stays on the forehead, and the cheeks stay on the cheeks.

Once the face is straightened, the system can use three simple, physics-based rules that were previously impossible to use on a moving face:

The Light Rule: It knows that skin facing the camera reflects light differently than skin turned away. It gives more weight to the parts of the face that are looking straight at the lens and ignores the parts that are turned away.
The Rhythm Rule: It knows that the heartbeat is a single wave. So, it checks if the forehead and both cheeks are pulsing in sync. If they aren't, it knows something is wrong.
The Teacher Rule: It uses a classic, old-school math trick (called POS) on the straightened forehead to create a "teacher" signal, which helps train the main computer brain to be more accurate.

The best part? This whole process doesn't require the computer to learn any new, heavy rules. It's like adding a pair of glasses to a camera; the camera's brain stays the same size, but now it can see clearly.

The results are pretty impressive. When they tested this on a dataset full of people turning their heads (called MMPD), the old methods got much worse as the head turned. For example, when the head turned more than 45 degrees, the error rate jumped by 1.60 times compared to looking straight ahead. With CanonicalPhys, that jump was cut down to just 1.33 times. Even better, for "mild" turns (between 15 and 30 degrees), the error barely changed at all, dropping from a 1.32 times increase to just 1.07 times.

They also tested this across different cameras and lighting conditions. When they trained the system on one dataset and tested it on others, CanonicalPhys beat the old methods in 13 out of 16 different scenarios. On one specific test (PURE), it reduced the error by 32%, and on another (MMPD), it cut the error by 20%.

However, the authors are careful to point out that this isn't a magic wand for every situation. If someone turns their head too far (more than 60 degrees), the system can't find the four key points anymore, and it just gives up, performing no better than the old methods. Also, if the face is moving wildly or if the lighting is weird, the system might get a little confused. But for the vast majority of real-world situations where people are just chatting or looking around, this "digital straightener" makes reading a heartbeat from a video much more reliable, proving that sometimes, the best way to solve a problem isn't to make the computer smarter, but to make the world a little easier for it to understand.

Technical Summary: CanonicalPhys

Problem Statement
Deep remote photoplethysmography (rPPG) models achieve sub-bpm heart-rate accuracy on frontal, stationary faces but suffer significant performance degradation when subjects exhibit head pose variations. The authors argue that this degradation is not merely a data-augmentation issue but a coordinate-structural problem. In standard image coordinates, a fixed pixel $(x, y)$ maps to different anatomical regions as the head rotates (e.g., drifting from forehead to temple). This instability prevents the application of three fundamental rPPG priors:

Dichromatic Reflection Model: The diffuse, pulse-carrying component of skin reflectance scales with the cosine of the angle between the skin normal and the camera axis, a property defined per anatomical location, not per pixel.
Pulse-Phase Invariance: The pulse phase should remain consistent across skin regions (e.g., forehead and cheeks), requiring a pose-stable definition of Regions of Interest (ROIs).
Chromaticity Projection (POS/CHROM): Classical methods project windowed RGB traces onto a chromaticity basis to suppress specular and illumination components, a process that is only well-posed on a fixed skin ROI.

Without a stable anatomy-to-pixel mapping, these priors cannot be effectively utilized, forcing deep learning backbones to absorb pose nuisance solely through model capacity.

Methodology: CanonicalPhys
The proposed CanonicalPhys framework addresses this by transforming the input video into a canonical face coordinate system before feature extraction. The pipeline consists of four stages, all wrapped around an unmodified FactorizePhys backbone (a state-of-the-art lightweight model using non-negative matrix factorization for spatial-temporal-channel attention).

Canonicalization via 4-Point Homography:
- A differentiable four-point homography is applied to each frame, mapping four facial anchors (outer left/right eye corners and left/right mouth corners) to fixed canonical target positions.
- This warp is computed using 2D landmarks (from MediaPipe) and solved via a closed-form linear system (DLT).
- Key Constraint: This module introduces no trainable parameters. It absorbs the dominant rigid, in-plane component of head pose, effectively restoring a stable anatomy-to-pixel mapping.
- Temporal stabilization is applied by computing the median homography over a video chunk to reduce jitter from noisy landmarks.
Physics-Informed Per-Pixel Weighting ( $w_{phys}$ ):
- Operating in the canonical frame, a per-pixel weight is computed based on the dichromatic reflection model.
- Using the canonical 3D normals of facial landmarks and the per-frame facial rotation matrix, the method calculates the cosine of the angle between the skin normal and the camera axis.
- A sigmoid function up-weights camera-facing pixels and down-weights grazing angles. This weight is applied multiplicatively to the canonical frame prior to temporal differencing.
Cross-ROI Temporal Consistency Loss:
- To enforce pulse-phase invariance, a self-supervised loss is applied to the backbone's feature maps.
- Three Gaussian soft masks (forehead, left cheek, right cheek) are defined in canonical coordinates.
- The loss minimizes the cosine distance between the temporal signals extracted from these three ROIs, ensuring the backbone learns spatially consistent pulse features.
POS Knowledge Distillation Loss:
- The classical POS algorithm is applied to the spatially averaged RGB trace of the canonical forehead (which is now pose-stable).
- The resulting pseudo-BVP serves as a teacher signal. A distillation loss (negative Pearson correlation) is computed between the network's prediction and this detached pseudo-BVP.
- This loss is applied only during training and adds no parameters.

Key Contributions

Coordinate Transformation: Introduction of a parameter-free, differentiable four-point homography that maps frames to a canonical system, isolating the rigid pose component before learned processing.
Canonical-Space Priors: Proposal of three new components enabled by this coordinate change:
- A physics-informed Lambertian weighting map.
- A cross-ROI temporal consistency loss enforcing phase invariance.
- Knowledge distillation from windowed POS applied to a stable forehead ROI.
- Crucially, none of these components add trainable parameters to the backbone.
Rigorous Evaluation: A pose-stratified analysis on the MMPD dataset that isolates pose effects from other distribution shifts, alongside full cross-dataset transfer evaluations.

Experimental Results

Pose-Stratified Analysis (MMPD):
- The baseline FactorizePhys shows a 1.60× degradation in Mean Absolute Error (MAE) from frontal ( $|yaw|<15^\circ$ ) to large-yaw ( $|yaw|\ge45^\circ$ ) frames.
- CanonicalPhys reduces this degradation ratio to 1.33×.
- The "mild-yaw" bin ( $15^\circ$ – $30^\circ$ ) degradation is flattened from 1.32× to 1.07×.
Cross-Dataset Transfer:
- Trained on UBFC-rPPG and tested on pose-rich targets, CanonicalPhys achieves a 32% MAE reduction on PURE and 20% on MMPD compared to FactorizePhys.
- It wins 13 out of 16 cells in the full 4×4 cross-dataset transfer matrix (UBFC, PURE, OBF, MMPD).
- On MMPD, the Pearson correlation nearly doubles (0.18 to 0.33), and seed variance is significantly reduced.
Ablation: The canonicalization warp alone accounts for the majority of the performance gain. The Lambertian weight and consistency loss provide further uniformity, particularly correcting anomalies in the moderate yaw range.

Significance and Claims
The paper claims that treating head pose as a coordinate-system problem rather than a data-augmentation challenge allows for the re-introduction of physical and physiological priors that are otherwise broken by pose variation.

Efficiency: The method achieves these gains without increasing the parameter count of the backbone, making it computationally efficient (wall-clock overhead <10%).
Generalization: The improvements transfer across different sensors and subject pools, suggesting the method addresses a fundamental geometric limitation rather than overfitting to specific dataset artifacts.
Limitations: The authors explicitly note that the method is pose-robust, not pose-invariant. Performance still degrades in extreme yaw ( $>60^\circ$ ) where landmarks fail, and the 2D homography is insufficient for significant out-of-plane rotation without a 3D mesh extension. Additionally, the auxiliary priors can over-regularize on highly motion-diverse training sets (e.g., MMPD training for PURE testing), suggesting the full "recipe" is best suited for compact training sets, while canonicalization alone is the most reliable component.

Technical Summary: CanonicalPhys

More like this